[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:04 dist-test-slave-tpsm krb5kdc[1380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:04 dist-test-slave-tpsm krb5kdc[1380](info): set up 2 sockets
Jun 19 13:00:04 dist-test-slave-tpsm krb5kdc[1380](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 13:00:05.604082 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.011s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:05 dist-test-slave-tpsm krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:05.772022 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:32789
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:06.097242 1393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:06.097517 1393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:06.097697 1393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:06.104988 1393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:06.105090 1393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:06.105182 1393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:06.105273 1393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:06.105345 1393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:06.114786 1393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:32789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:06.116098 1393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:06.117441 1393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:06.135290 1398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:06.135515 1399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:06.135666 1401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:06.136015 1393 server_base.cc:1034] running on GCE node
I20260619 13:00:06.137028 1393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:06.138566 1393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:06.139780 1393 hybrid_clock.cc:648] HybridClock initialized: now 1781874006139693 us; error 99 us; skew 500 ppm
I20260619 13:00:06.140095 1393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:06 dist-test-slave-tpsm krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874006, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:06.143108 1393 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:06.144788 1393 webserver.cc:460] Webserver started at http://127.1.87.126:42803/ using document root <none> and password file <none>
I20260619 13:00:06.145483 1393 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:06.145607 1393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:06.145915 1393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:06.149287 1393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "62b2caf05be645f3a1d0ecfc29088425"
format_stamp: "Formatted at 2026-06-19 13:00:06 on dist-test-slave-tpsm"
server_key: "be6d4866f141d8fbbbf68aa20e6c8945"
server_key_iv: "12714098b09bfde112c60045b1526102"
server_key_version: "encryptionkey@0"
I20260619 13:00:06.150058 1393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "62b2caf05be645f3a1d0ecfc29088425"
format_stamp: "Formatted at 2026-06-19 13:00:06 on dist-test-slave-tpsm"
server_key: "be6d4866f141d8fbbbf68aa20e6c8945"
server_key_iv: "12714098b09bfde112c60045b1526102"
server_key_version: "encryptionkey@0"
I20260619 13:00:06.155005 1393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 13:00:06.158295 1408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:06.159650 1393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:00:06.159821 1393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "62b2caf05be645f3a1d0ecfc29088425"
format_stamp: "Formatted at 2026-06-19 13:00:06 on dist-test-slave-tpsm"
server_key: "be6d4866f141d8fbbbf68aa20e6c8945"
server_key_iv: "12714098b09bfde112c60045b1526102"
server_key_version: "encryptionkey@0"
I20260619 13:00:06.159977 1393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:06.174289 1393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:06.178726 1393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:06.179005 1393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:06.198277 1393 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:32789
I20260619 13:00:06.198300 1459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:32789 every 8 connection(s)
I20260619 13:00:06.199664 1393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:06.203848 1460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:06.208197 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 1393
I20260619 13:00:06.208500 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:06.210321 1373 external_mini_cluster.cc:1428] Setting key 9447624cdb6bf2d191dca0882446a36f
I20260619 13:00:06.214505 1460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425: Bootstrap starting.
I20260619 13:00:06.218571 1460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:06.219733 1460 log.cc:824] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:06.222656 1460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425: No bootstrap required, opened a new log
I20260619 13:00:06.233605 1460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b2caf05be645f3a1d0ecfc29088425" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 32789 } }
I20260619 13:00:06.234040 1460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:06.234164 1460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b2caf05be645f3a1d0ecfc29088425, State: Initialized, Role: FOLLOWER
I20260619 13:00:06.234889 1460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b2caf05be645f3a1d0ecfc29088425" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 32789 } }
I20260619 13:00:06.235200 1460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:06.235419 1460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:06.235630 1460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:06.237824 1460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b2caf05be645f3a1d0ecfc29088425" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 32789 } }
I20260619 13:00:06.238499 1460 leader_election.cc:302] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62b2caf05be645f3a1d0ecfc29088425; no voters:
I20260619 13:00:06.239136 1460 leader_election.cc:288] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:06.239346 1465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:06.240644 1465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [term 1 LEADER]: Becoming Leader. State: Replica: 62b2caf05be645f3a1d0ecfc29088425, State: Running, Role: LEADER
Jun 19 13:00:06 dist-test-slave-tpsm krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:06.241235 1465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b2caf05be645f3a1d0ecfc29088425" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 32789 } }
I20260619 13:00:06.241534 1460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:06.245424 1466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62b2caf05be645f3a1d0ecfc29088425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b2caf05be645f3a1d0ecfc29088425" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 32789 } } }
I20260619 13:00:06.245676 1467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62b2caf05be645f3a1d0ecfc29088425. Latest consensus state: current_term: 1 leader_uuid: "62b2caf05be645f3a1d0ecfc29088425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b2caf05be645f3a1d0ecfc29088425" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 32789 } } }
I20260619 13:00:06.246443 1467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:06.246621 1466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:06.258406 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:06.214153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.1.87.126:32789)
0619 13:00:06.214988 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:06.215005 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:06.216754 (+ 1749us) server_negotiation.cc:407] Connection header received
0619 13:00:06.225853 (+ 9099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:06.225884 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:06.226436 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:06.226941 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:06.229720 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:06.232225 (+ 2505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:06.234097 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:06.235240 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:06.235465 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:06.243153 (+ 7688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:06.243168 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:06.243193 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:06.243256 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:06.248648 (+ 5392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:06.250051 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:06.250063 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:06.250074 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:06.250349 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:06.251024 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:06.251031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:06.251037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:06.251939 (+ 902us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:06.255661 (+ 3722us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:06.256012 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 13:00:06.256580 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":381,"threads_started":1}
I20260619 13:00:06.262032 1474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:06.267915 1474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:06.276769 1474 catalog_manager.cc:1347] Generated new cluster ID: e70939b8d99f438bb6e46acc27055ac8
I20260619 13:00:06.276901 1474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:06.295166 1474 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:06.296849 1474 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:06.305214 1474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62b2caf05be645f3a1d0ecfc29088425: Generated new TSK 0
I20260619 13:00:06.306022 1474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:00:06.354055 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:32789
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:06.707410 1486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:06.708078 1486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:06.708194 1486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:06.715325 1486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:06.715428 1486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:06.715543 1486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:06.725603 1486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:32789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:06.726817 1486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:06.728091 1486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:06.741027 1491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:06.741080 1492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:06.741338 1494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:06.742048 1486 server_base.cc:1034] running on GCE node
I20260619 13:00:06.742662 1486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:06.743696 1486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:06.744889 1486 hybrid_clock.cc:648] HybridClock initialized: now 1781874006744835 us; error 59 us; skew 500 ppm
I20260619 13:00:06.745183 1486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:06 dist-test-slave-tpsm krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874006, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:06.748435 1486 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:00:06.749756 1486 webserver.cc:460] Webserver started at http://127.1.87.65:46523/ using document root <none> and password file <none>
I20260619 13:00:06.750360 1486 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:06.750454 1486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:06.751091 1486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:06.753897 1486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "e3e8d5f5016a4c939fc1111889cc8dad"
format_stamp: "Formatted at 2026-06-19 13:00:06 on dist-test-slave-tpsm"
server_key: "830d1e4117af4a1cca926f772d084db3"
server_key_iv: "64322e493bd0fc564d4afdcb404e5976"
server_key_version: "encryptionkey@0"
I20260619 13:00:06.754561 1486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "e3e8d5f5016a4c939fc1111889cc8dad"
format_stamp: "Formatted at 2026-06-19 13:00:06 on dist-test-slave-tpsm"
server_key: "830d1e4117af4a1cca926f772d084db3"
server_key_iv: "64322e493bd0fc564d4afdcb404e5976"
server_key_version: "encryptionkey@0"
I20260619 13:00:06.759945 1486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 13:00:06.763351 1501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:06.764868 1486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:00:06.765034 1486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "e3e8d5f5016a4c939fc1111889cc8dad"
format_stamp: "Formatted at 2026-06-19 13:00:06 on dist-test-slave-tpsm"
server_key: "830d1e4117af4a1cca926f772d084db3"
server_key_iv: "64322e493bd0fc564d4afdcb404e5976"
server_key_version: "encryptionkey@0"
I20260619 13:00:06.765234 1486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:06.789811 1486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:06.794057 1486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:06.794345 1486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:06.795635 1486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:06.796929 1486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:06.797030 1486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:06.797181 1486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:06.797254 1486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:06.831770 1486 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:36773
I20260619 13:00:06.831821 1613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:36773 every 8 connection(s)
I20260619 13:00:06.833024 1486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:06.842559 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 1486
I20260619 13:00:06.842834 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:06.843480 1373 external_mini_cluster.cc:1428] Setting key a927346b3d856036e0b8455d07226799
Jun 19 13:00:06 dist-test-slave-tpsm krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874006, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:06.852053 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:06.835422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:55599 (local address 127.1.87.126:32789)
0619 13:00:06.835851 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:06.835855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:06.836649 (+ 794us) server_negotiation.cc:407] Connection header received
0619 13:00:06.838375 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:06.838381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:06.838463 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:06.838579 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:06.840002 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:06.840728 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:06.842271 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:06.842645 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:06.842822 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:06.846237 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:06.846242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:06.846245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:06.846272 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:06.849237 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:06.849924 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:06.849934 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:06.849937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:06.850043 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:06.850445 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:06.850450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:06.850454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:06.850685 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:06.850831 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:06.851507 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 13:00:06.851708 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
I20260619 13:00:06.853018 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:06.835852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:32789 (local address 127.1.87.65:55599)
0619 13:00:06.836503 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:06.836538 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:06.838123 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:06.838791 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:06.838799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:06.839285 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:06.839850 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:06.839866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:06.840871 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:06.840881 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:06.842140 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:06.842151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:06.842912 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:06.842919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:06.843131 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:06.843885 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:06.843906 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:06.846032 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:06.849380 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:06.849388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:06.849404 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:06.849802 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:06.850146 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:06.850151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:06.850155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:06.850331 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:06.850811 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:06.850819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:06.851118 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 13:00:06.851403 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 13:00:06.851901 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":259,"threads_started":1}
I20260619 13:00:06.854815 1614 heartbeater.cc:347] Connected to a master server at 127.1.87.126:32789
I20260619 13:00:06.855167 1614 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:06.856103 1614 heartbeater.cc:511] Master 127.1.87.126:32789 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
I20260619 13:00:06.858228 1425 ts_manager.cc:194] Registered new tserver with Master: e3e8d5f5016a4c939fc1111889cc8dad (127.1.87.65:36773)
I20260619 13:00:06.859645 1425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:55599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:00:06.878175 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:32789
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:07.196019 1620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:07.196278 1620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:07.196445 1620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:07.203507 1620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:07.203603 1620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:07.203775 1620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:07.213147 1620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:32789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:07.214282 1620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:07.215950 1620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:07.229558 1625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:07.229509 1626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:07.229590 1628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:07.230629 1620 server_base.cc:1034] running on GCE node
I20260619 13:00:07.231159 1620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:07.232080 1620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:07.233264 1620 hybrid_clock.cc:648] HybridClock initialized: now 1781874007233214 us; error 47 us; skew 500 ppm
I20260619 13:00:07.233522 1620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:07 dist-test-slave-tpsm krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:07.240442 1620 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:00:07.241916 1620 webserver.cc:460] Webserver started at http://127.1.87.66:39431/ using document root <none> and password file <none>
I20260619 13:00:07.242547 1620 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:07.242642 1620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:07.243000 1620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:07.246057 1620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "b416c1387a954dd1ac92da7eec085d62"
format_stamp: "Formatted at 2026-06-19 13:00:07 on dist-test-slave-tpsm"
server_key: "71a61c7b9e0dca80b745e8f1538d5c8e"
server_key_iv: "b0c0583f44168ff37720539da2298283"
server_key_version: "encryptionkey@0"
I20260619 13:00:07.246778 1620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "b416c1387a954dd1ac92da7eec085d62"
format_stamp: "Formatted at 2026-06-19 13:00:07 on dist-test-slave-tpsm"
server_key: "71a61c7b9e0dca80b745e8f1538d5c8e"
server_key_iv: "b0c0583f44168ff37720539da2298283"
server_key_version: "encryptionkey@0"
I20260619 13:00:07.252048 1620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:00:07.255435 1635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:07.256807 1620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:00:07.256935 1620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "b416c1387a954dd1ac92da7eec085d62"
format_stamp: "Formatted at 2026-06-19 13:00:07 on dist-test-slave-tpsm"
server_key: "71a61c7b9e0dca80b745e8f1538d5c8e"
server_key_iv: "b0c0583f44168ff37720539da2298283"
server_key_version: "encryptionkey@0"
I20260619 13:00:07.257097 1620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:07.271407 1620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:07.276173 1620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:07.276415 1620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:07.277297 1620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:07.278532 1620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:07.278602 1620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:07.278683 1620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:07.278739 1620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:07.313335 1620 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:44837
I20260619 13:00:07.313370 1747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:44837 every 8 connection(s)
I20260619 13:00:07.314553 1620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:07.324313 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 1620
I20260619 13:00:07.324514 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:07.324916 1373 external_mini_cluster.cc:1428] Setting key 5b8c3651b427e0aa9d6fc2db79a776a4
Jun 19 13:00:07 dist-test-slave-tpsm krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:07.335803 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:07.317028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:33091 (local address 127.1.87.126:32789)
0619 13:00:07.317209 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:07.317213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:07.318287 (+ 1074us) server_negotiation.cc:407] Connection header received
0619 13:00:07.320686 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:07.320692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:07.320782 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:07.320895 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:07.322794 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.323549 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.325167 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.325581 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.325723 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:07.329731 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:07.329736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:07.329739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:07.329769 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:07.332589 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.333348 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.333356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.333360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.333506 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.333999 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.334004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.334009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.334303 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:07.334461 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:07.335154 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 13:00:07.335409 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:00:07.336875 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:07.317487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:32789 (local address 127.1.87.66:33091)
0619 13:00:07.318137 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:07.318181 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:07.320397 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:07.321120 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:07.321128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:07.321830 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:07.322627 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:07.322646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.323709 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.323713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:07.325018 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:07.325032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.325819 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.325827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:07.326044 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:07.326730 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:07.326761 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:07.329501 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:07.332748 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:07.332757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:07.332774 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:07.333213 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:07.333633 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:07.333639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:07.333643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:07.333850 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:07.334437 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:07.334445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:07.334780 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 13:00:07.335066 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 13:00:07.335609 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":251,"threads_started":1}
I20260619 13:00:07.338416 1748 heartbeater.cc:347] Connected to a master server at 127.1.87.126:32789
I20260619 13:00:07.338742 1748 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:07.339582 1748 heartbeater.cc:511] Master 127.1.87.126:32789 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:00:07.341127 1425 ts_manager.cc:194] Registered new tserver with Master: b416c1387a954dd1ac92da7eec085d62 (127.1.87.66:44837)
I20260619 13:00:07.341955 1425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:33091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:00:07.361665 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:32789
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:07.683964 1753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:07.684229 1753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:07.684398 1753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:07.691447 1753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:07.691550 1753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:07.691782 1753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:07.701531 1753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:32789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:07.702780 1753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:07.704038 1753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:07.715984 1758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:07.716063 1759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:07.716363 1761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:07.716995 1753 server_base.cc:1034] running on GCE node
I20260619 13:00:07.717527 1753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:07.718360 1753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:07.719591 1753 hybrid_clock.cc:648] HybridClock initialized: now 1781874007719551 us; error 51 us; skew 500 ppm
I20260619 13:00:07.719956 1753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:07 dist-test-slave-tpsm krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:07.723500 1753 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:00:07.724891 1753 webserver.cc:460] Webserver started at http://127.1.87.67:42311/ using document root <none> and password file <none>
I20260619 13:00:07.725557 1753 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:07.725710 1753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:07.726016 1753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:07.728883 1753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "e902a5981cf54f66a4f37fc8cbc2a46b"
format_stamp: "Formatted at 2026-06-19 13:00:07 on dist-test-slave-tpsm"
server_key: "c5d451e30ca4f87d3a1d1ffdfdb1e8c8"
server_key_iv: "96f4c047f0b7314d08ae880669724466"
server_key_version: "encryptionkey@0"
I20260619 13:00:07.729641 1753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "e902a5981cf54f66a4f37fc8cbc2a46b"
format_stamp: "Formatted at 2026-06-19 13:00:07 on dist-test-slave-tpsm"
server_key: "c5d451e30ca4f87d3a1d1ffdfdb1e8c8"
server_key_iv: "96f4c047f0b7314d08ae880669724466"
server_key_version: "encryptionkey@0"
I20260619 13:00:07.734396 1753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:00:07.737797 1768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:07.739104 1753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:00:07.739284 1753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "e902a5981cf54f66a4f37fc8cbc2a46b"
format_stamp: "Formatted at 2026-06-19 13:00:07 on dist-test-slave-tpsm"
server_key: "c5d451e30ca4f87d3a1d1ffdfdb1e8c8"
server_key_iv: "96f4c047f0b7314d08ae880669724466"
server_key_version: "encryptionkey@0"
I20260619 13:00:07.739571 1753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:07.760425 1753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:07.765259 1753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:07.765553 1753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:07.766466 1753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:07.767947 1753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:07.768050 1753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:07.768187 1753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:07.768255 1753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:07.803215 1753 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:46711
I20260619 13:00:07.803299 1880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:46711 every 8 connection(s)
I20260619 13:00:07.804471 1753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:07.808305 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 1753
I20260619 13:00:07.808494 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:07.809000 1373 external_mini_cluster.cc:1428] Setting key effe7bc9268ed257103735d7d79bc2e2
Jun 19 13:00:07 dist-test-slave-tpsm krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:07.823608 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:07.807269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:57995 (local address 127.1.87.126:32789)
0619 13:00:07.807493 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:07.807499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:07.808762 (+ 1263us) server_negotiation.cc:407] Connection header received
0619 13:00:07.810762 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:07.810767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:07.810881 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:07.810987 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:07.812429 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.813175 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.814656 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.815048 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.815184 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:07.818367 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:07.818371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:07.818375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:07.818403 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:07.820795 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.821460 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.821471 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.821474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.821585 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.821989 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.821994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.821997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.822245 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:07.822391 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:07.823035 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 13:00:07.823236 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 13:00:07.824702 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:07.807766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:32789 (local address 127.1.87.67:57995)
0619 13:00:07.808601 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:07.808655 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:07.810522 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:07.811216 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:07.811224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:07.811759 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:07.812273 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:07.812288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.813331 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.813335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:07.814513 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:07.814524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.815265 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.815274 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:07.815522 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:07.816178 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:07.816203 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:07.818168 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:07.820924 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:07.820932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:07.820946 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:07.821338 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:07.821698 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:07.821703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:07.821707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:07.821875 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:07.822366 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:07.822374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:07.822669 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 13:00:07.822915 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 13:00:07.823430 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":327,"threads_started":1}
I20260619 13:00:07.825949 1881 heartbeater.cc:347] Connected to a master server at 127.1.87.126:32789
I20260619 13:00:07.826264 1881 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:07.826941 1881 heartbeater.cc:511] Master 127.1.87.126:32789 requested a full tablet report, sending...
I20260619 13:00:07.828732 1425 ts_manager.cc:194] Registered new tserver with Master: e902a5981cf54f66a4f37fc8cbc2a46b (127.1.87.67:46711)
I20260619 13:00:07.829515 1425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:57995
I20260619 13:00:07.836365 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:00:07 dist-test-slave-tpsm krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260619 13:00:07.862283 1614 heartbeater.cc:503] Master 127.1.87.126:32789 was elected leader, sending a full tablet report...
Jun 19 13:00:07 dist-test-slave-tpsm krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:07.879488 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:07.864661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.1.87.65:36773)
0619 13:00:07.865244 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:07.865251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:07.865275 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:00:07.865368 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:07.865374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:07.865565 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:07.865700 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:07.867032 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.868162 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.869606 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.870177 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.870274 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:07.874339 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:07.874348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:07.874362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:07.874395 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:07.877172 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.877759 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.877766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.877768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.877886 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.878211 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.878214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.878216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.878506 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:07.878708 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:07.878941 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 13:00:07.879116 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":296,"threads_started":1}
W20260619 13:00:07.880688 1529 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52592
Jun 19 13:00:07 dist-test-slave-tpsm krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:00:07 dist-test-slave-tpsm krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:07.922086 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:07.908480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.1.87.65:36773)
0619 13:00:07.908722 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:07.908730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:07.908853 (+ 123us) server_negotiation.cc:407] Connection header received
0619 13:00:07.908987 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:07.908991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:07.909097 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:07.909193 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:07.910466 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.911423 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.912638 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.913098 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.913204 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:07.917326 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:07.917331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:07.917334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:07.917365 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:07.919687 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.920223 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.920226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.920228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.920368 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.920773 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.920777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.920781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.921030 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:07.921271 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:07.921568 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 13:00:07.921773 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 13:00:07.923615 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 1486
I20260619 13:00:07.935792 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:07.924465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.1.87.65:36773)
0619 13:00:07.924636 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:07.924642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:07.924985 (+ 343us) server_negotiation.cc:407] Connection header received
0619 13:00:07.925155 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:07.925158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:07.925209 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:07.925307 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:07.926372 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.927042 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.928287 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:07.928687 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:07.928810 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:07.930826 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:07.930835 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:07.930839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:07.930866 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:07.933359 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.933996 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.934001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.934006 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.934131 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:07.934474 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:07.934480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:07.934483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:07.934728 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:07.934964 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:07.935246 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 13:00:07.935461 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 13:00:07.936440 1609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:08.018158 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 1486
I20260619 13:00:08.034705 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 1620
Jun 19 13:00:08 dist-test-slave-tpsm krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:00:08.051052 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:08.035704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.1.87.66:44837)
0619 13:00:08.036192 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:08.036200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:08.036223 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:00:08.036279 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:08.036284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:08.036476 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:08.036600 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:08.037705 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:08.038680 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:08.040038 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:08.040545 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:08.040638 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:08.044833 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:08.044844 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:08.044858 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:08.044890 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:08.048234 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:08.048962 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:08.048969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:08.048971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:08.049083 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:08.049481 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:08.049491 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:08.049493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:08.049813 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:08.050114 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:08.050441 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 13:00:08.050716 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":247,"threads_started":1}
I20260619 13:00:08.052469 1743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:08.132797 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 1620
I20260619 13:00:08.154343 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 1753
Jun 19 13:00:08 dist-test-slave-tpsm krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:00:08.169888 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:08.155396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.1.87.67:46711)
0619 13:00:08.155992 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:08.156000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:08.156027 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:00:08.156093 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:08.156099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:08.156305 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:08.156477 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:08.157593 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:08.158613 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:08.159897 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:08.160455 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:08.160559 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:08.164515 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:08.164534 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:08.164549 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:08.164582 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:08.167258 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:08.168088 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:08.168095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:08.168098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:08.168221 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:08.168539 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:08.168542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:08.168544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:08.168869 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:08.169100 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:08.169320 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 13:00:08.169508 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":300,"threads_started":1}
I20260619 13:00:08.171120 1876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:08.251784 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 1753
I20260619 13:00:08.268723 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 1393
I20260619 13:00:08.269841 1455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:08.369211 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 1393
2026-06-19T13:00:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3862 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:08 dist-test-slave-tpsm krb5kdc[1902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:08 dist-test-slave-tpsm krb5kdc[1902](info): set up 2 sockets
Jun 19 13:00:08 dist-test-slave-tpsm krb5kdc[1902](info): commencing operation
krb5kdc: starting...
W20260619 13:00:09.451687 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:09 dist-test-slave-tpsm krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:09.620330 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:34747
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:34747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:09.943380 1915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:09.943717 1915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:09.943892 1915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:09.951025 1915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:09.951192 1915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:09.951303 1915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:09.951423 1915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:09.951512 1915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:09.961093 1915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:34747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:34747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:09.962464 1915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:09.963795 1915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:09.976142 1920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:09.976207 1921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:09.976300 1923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:09.976944 1915 server_base.cc:1034] running on GCE node
I20260619 13:00:09.977715 1915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:09.979156 1915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:09.980393 1915 hybrid_clock.cc:648] HybridClock initialized: now 1781874009980312 us; error 76 us; skew 500 ppm
I20260619 13:00:09.980741 1915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:09 dist-test-slave-tpsm krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:09.983904 1915 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:09.985272 1915 webserver.cc:460] Webserver started at http://127.1.87.126:46023/ using document root <none> and password file <none>
I20260619 13:00:09.985989 1915 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:09.986089 1915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:09.986344 1915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:09.989423 1915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "165d9b804c414190b4d1df915d1e4608"
format_stamp: "Formatted at 2026-06-19 13:00:09 on dist-test-slave-tpsm"
server_key: "d62974496224d48a1e78f07e3f916c77"
server_key_iv: "7174412c1043c863ef4144355f74f6d0"
server_key_version: "encryptionkey@0"
I20260619 13:00:09.990192 1915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "165d9b804c414190b4d1df915d1e4608"
format_stamp: "Formatted at 2026-06-19 13:00:09 on dist-test-slave-tpsm"
server_key: "d62974496224d48a1e78f07e3f916c77"
server_key_iv: "7174412c1043c863ef4144355f74f6d0"
server_key_version: "encryptionkey@0"
I20260619 13:00:09.995463 1915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 13:00:09.998965 1930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:10.000377 1915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:10.000552 1915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "165d9b804c414190b4d1df915d1e4608"
format_stamp: "Formatted at 2026-06-19 13:00:09 on dist-test-slave-tpsm"
server_key: "d62974496224d48a1e78f07e3f916c77"
server_key_iv: "7174412c1043c863ef4144355f74f6d0"
server_key_version: "encryptionkey@0"
I20260619 13:00:10.000773 1915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:10.017612 1915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:10.022156 1915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:10.022465 1915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:10.041597 1915 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:34747
I20260619 13:00:10.041630 1981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:34747 every 8 connection(s)
I20260619 13:00:10.043334 1915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:10.047147 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:10.047262 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 1915
I20260619 13:00:10.047554 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:10.048207 1373 external_mini_cluster.cc:1428] Setting key fc035e63480efea03452da5415bb465d
I20260619 13:00:10.061578 1982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608: Bootstrap starting.
Jun 19 13:00:10 dist-test-slave-tpsm krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:10.065176 1982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:10.066385 1982 log.cc:824] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:10.069499 1982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608: No bootstrap required, opened a new log
I20260619 13:00:10.073170 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:10.050148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.1.87.126:34747)
0619 13:00:10.050919 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:10.050930 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:10.050970 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:00:10.053122 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:10.053144 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:10.053550 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:10.053890 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:10.055687 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:10.056966 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:10.058882 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:10.059674 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:10.059819 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:10.064889 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:10.064899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:10.064914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:10.064959 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:10.068687 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:10.069460 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:10.069475 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:10.069481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:10.069704 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:10.070107 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:10.070115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:10.070119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:10.070670 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:10.071008 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:10.071588 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 13:00:10.072076 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":347,"threads_started":1}
I20260619 13:00:10.077980 1982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d9b804c414190b4d1df915d1e4608" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34747 } }
I20260619 13:00:10.078320 1982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:10.078442 1982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165d9b804c414190b4d1df915d1e4608, State: Initialized, Role: FOLLOWER
I20260619 13:00:10.079026 1982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d9b804c414190b4d1df915d1e4608" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34747 } }
I20260619 13:00:10.079250 1982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:10.079428 1982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:10.079610 1982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:10.081975 1982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d9b804c414190b4d1df915d1e4608" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34747 } }
I20260619 13:00:10.082386 1982 leader_election.cc:302] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165d9b804c414190b4d1df915d1e4608; no voters:
I20260619 13:00:10.082872 1982 leader_election.cc:288] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:10.083021 1987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:10.083680 1987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [term 1 LEADER]: Becoming Leader. State: Replica: 165d9b804c414190b4d1df915d1e4608, State: Running, Role: LEADER
I20260619 13:00:10.084183 1987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d9b804c414190b4d1df915d1e4608" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34747 } }
I20260619 13:00:10.084856 1982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:10.086891 1988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165d9b804c414190b4d1df915d1e4608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d9b804c414190b4d1df915d1e4608" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34747 } } }
I20260619 13:00:10.087287 1988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:10.088364 1989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165d9b804c414190b4d1df915d1e4608. Latest consensus state: current_term: 1 leader_uuid: "165d9b804c414190b4d1df915d1e4608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d9b804c414190b4d1df915d1e4608" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34747 } } }
I20260619 13:00:10.088754 1989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:10.090091 1994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:10.096392 1994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:10.106117 1994 catalog_manager.cc:1347] Generated new cluster ID: b5455231c1da44afb600fa36de5a2ab3
I20260619 13:00:10.106254 1994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:10.114512 1994 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:10.116465 1994 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:10.133888 1994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 165d9b804c414190b4d1df915d1e4608: Generated new TSK 0
I20260619 13:00:10.134979 1994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:00:10.181028 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:34747
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:10.502800 2008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:10.503140 2008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:10.503293 2008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:10.510854 2008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:10.511030 2008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:10.511199 2008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:10.521364 2008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:34747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:10.522753 2008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:10.524165 2008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:10.539346 2016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:10.539376 2013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:10.539402 2014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:10.541249 2008 server_base.cc:1034] running on GCE node
I20260619 13:00:10.542132 2008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:10.543154 2008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:10.544420 2008 hybrid_clock.cc:648] HybridClock initialized: now 1781874010544368 us; error 68 us; skew 500 ppm
I20260619 13:00:10.544790 2008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:10 dist-test-slave-tpsm krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:10.548174 2008 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:00:10.549723 2008 webserver.cc:460] Webserver started at http://127.1.87.65:34015/ using document root <none> and password file <none>
I20260619 13:00:10.550424 2008 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:10.550570 2008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:10.550891 2008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:10.553859 2008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "185273f370fb45189c36c6fa466385d7"
format_stamp: "Formatted at 2026-06-19 13:00:10 on dist-test-slave-tpsm"
server_key: "8584333c8b8ef45c2709040a56a5f5ae"
server_key_iv: "324dcbf9c27b93948db816da6202e87d"
server_key_version: "encryptionkey@0"
I20260619 13:00:10.554661 2008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "185273f370fb45189c36c6fa466385d7"
format_stamp: "Formatted at 2026-06-19 13:00:10 on dist-test-slave-tpsm"
server_key: "8584333c8b8ef45c2709040a56a5f5ae"
server_key_iv: "324dcbf9c27b93948db816da6202e87d"
server_key_version: "encryptionkey@0"
I20260619 13:00:10.560141 2008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:00:10.563807 2023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:10.565236 2008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:00:10.565428 2008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "185273f370fb45189c36c6fa466385d7"
format_stamp: "Formatted at 2026-06-19 13:00:10 on dist-test-slave-tpsm"
server_key: "8584333c8b8ef45c2709040a56a5f5ae"
server_key_iv: "324dcbf9c27b93948db816da6202e87d"
server_key_version: "encryptionkey@0"
I20260619 13:00:10.565661 2008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:10.584573 2008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:10.588974 2008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:10.589278 2008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:10.590255 2008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:10.591714 2008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:10.591812 2008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:10.591950 2008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:10.592022 2008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:10.624616 2008 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:42995
I20260619 13:00:10.624701 2135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:42995 every 8 connection(s)
I20260619 13:00:10.625954 2008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:10.629362 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 2008
I20260619 13:00:10.629624 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:10.630393 1373 external_mini_cluster.cc:1428] Setting key afae1916a1a4de760d232e207c8fdf84
Jun 19 13:00:10 dist-test-slave-tpsm krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
I20260619 13:00:10.647234 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:10.628540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:34283 (local address 127.1.87.126:34747)
0619 13:00:10.629014 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:10.629018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:10.630075 (+ 1057us) server_negotiation.cc:407] Connection header received
0619 13:00:10.632151 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:10.632154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:10.632254 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:10.632379 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:10.633918 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:10.635164 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:10.636906 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:10.637290 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:10.637387 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:10.641633 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:10.641638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:10.641642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:10.641674 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:10.644222 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:10.644964 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:10.644969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:10.644973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:10.645098 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:10.645514 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:10.645518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:10.645522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:10.645806 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:10.645955 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:10.646657 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 13:00:10.646875 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":284,"threads_started":1}
I20260619 13:00:10.648214 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:10.629041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:34747 (local address 127.1.87.65:34283)
0619 13:00:10.629929 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:10.629964 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:10.631853 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:10.632621 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:10.632629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:10.633123 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:10.633744 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:10.633762 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:10.635391 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:10.635395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:10.636757 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:10.636774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:10.637566 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:10.637577 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:10.637930 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:10.638672 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:10.638711 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:10.641406 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:10.644371 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:10.644380 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:10.644404 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:10.644832 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:10.645207 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:10.645212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:10.645216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:10.645395 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:10.645930 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:10.645938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:10.646295 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 13:00:10.646553 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 13:00:10.647091 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":382,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
I20260619 13:00:10.649770 2136 heartbeater.cc:347] Connected to a master server at 127.1.87.126:34747
I20260619 13:00:10.650107 2136 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:10.651007 2136 heartbeater.cc:511] Master 127.1.87.126:34747 requested a full tablet report, sending...
I20260619 13:00:10.653337 1947 ts_manager.cc:194] Registered new tserver with Master: 185273f370fb45189c36c6fa466385d7 (127.1.87.65:42995)
I20260619 13:00:10.654860 1947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:34283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:00:10.668979 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:34747
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:10.984036 2142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:10.984308 2142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:10.984475 2142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:10.992096 2142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:10.992223 2142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:10.992396 2142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:11.002147 2142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:34747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:11.003482 2142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:11.004909 2142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:11.017839 2148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:11.017863 2150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:11.017962 2147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:11.018473 2142 server_base.cc:1034] running on GCE node
I20260619 13:00:11.019243 2142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:11.020211 2142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:11.021437 2142 hybrid_clock.cc:648] HybridClock initialized: now 1781874011021388 us; error 32 us; skew 500 ppm
I20260619 13:00:11.021802 2142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:11 dist-test-slave-tpsm krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:11.025207 2142 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:00:11.026806 2142 webserver.cc:460] Webserver started at http://127.1.87.66:43305/ using document root <none> and password file <none>
I20260619 13:00:11.027585 2142 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:11.027733 2142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:11.028043 2142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:11.031119 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "aeaf88fc40db4542b3b3ca2650609302"
format_stamp: "Formatted at 2026-06-19 13:00:11 on dist-test-slave-tpsm"
server_key: "beeef358cd06b50768f15ed505a3ae0d"
server_key_iv: "cb35b4608e32372b7e14e5a0c8112286"
server_key_version: "encryptionkey@0"
I20260619 13:00:11.032032 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "aeaf88fc40db4542b3b3ca2650609302"
format_stamp: "Formatted at 2026-06-19 13:00:11 on dist-test-slave-tpsm"
server_key: "beeef358cd06b50768f15ed505a3ae0d"
server_key_iv: "cb35b4608e32372b7e14e5a0c8112286"
server_key_version: "encryptionkey@0"
I20260619 13:00:11.037286 2142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:00:11.040771 2157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:11.042341 2142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 13:00:11.042506 2142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "aeaf88fc40db4542b3b3ca2650609302"
format_stamp: "Formatted at 2026-06-19 13:00:11 on dist-test-slave-tpsm"
server_key: "beeef358cd06b50768f15ed505a3ae0d"
server_key_iv: "cb35b4608e32372b7e14e5a0c8112286"
server_key_version: "encryptionkey@0"
I20260619 13:00:11.042724 2142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:11.066742 2142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:11.070912 2142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:11.071174 2142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:11.072196 2142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:11.073442 2142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:11.073513 2142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:11.073598 2142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:11.073645 2142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:11.106791 2142 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:37063
I20260619 13:00:11.106837 2269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:37063 every 8 connection(s)
I20260619 13:00:11.108114 2142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:11.116096 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 2142
I20260619 13:00:11.116320 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:11.116742 1373 external_mini_cluster.cc:1428] Setting key 94c4d972e72c9f2d42db74ff2f898427
Jun 19 13:00:11 dist-test-slave-tpsm krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:11.129328 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.110702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:48477 (local address 127.1.87.126:34747)
0619 13:00:11.110889 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.110893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.111992 (+ 1099us) server_negotiation.cc:407] Connection header received
0619 13:00:11.113780 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.113785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.113879 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.113991 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:11.115514 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.116278 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.118013 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.118568 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.118692 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.122919 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:11.122928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:11.122936 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:11.122982 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:11.125720 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.126697 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.126701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.126703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.126841 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.127447 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.127451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.127453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.127824 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:11.127947 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.128895 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.129075 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:00:11.131048 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.111022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:34747 (local address 127.1.87.66:48477)
0619 13:00:11.111819 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:11.111858 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:11.113515 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:11.114239 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:11.114247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:11.114748 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:11.115287 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.115304 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.116436 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.116442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.117870 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.117882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.118801 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.118808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.119081 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.119850 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:11.119891 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:11.122665 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:11.125872 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:11.125884 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:11.125906 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:11.126559 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:11.126976 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:11.126991 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:11.126998 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:11.127268 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:11.127967 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:11.127978 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:11.128482 (+ 504us) client_negotiation.cc:769] Sending connection context
0619 13:00:11.128807 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 13:00:11.129428 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":365,"threads_started":1}
I20260619 13:00:11.132601 2270 heartbeater.cc:347] Connected to a master server at 127.1.87.126:34747
I20260619 13:00:11.132946 2270 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:11.133744 2270 heartbeater.cc:511] Master 127.1.87.126:34747 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:00:11.135366 1947 ts_manager.cc:194] Registered new tserver with Master: aeaf88fc40db4542b3b3ca2650609302 (127.1.87.66:37063)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
I20260619 13:00:11.136205 1947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:48477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:00:11.155990 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:34747
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:11.474799 2275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:11.475113 2275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:11.475301 2275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:11.483959 2275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:11.484097 2275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:11.484290 2275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:11.496537 2275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:34747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:11.497982 2275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:11.499306 2275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:11.513388 2283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:11.513468 2281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:11.513427 2280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:11.514947 2275 server_base.cc:1034] running on GCE node
I20260619 13:00:11.515702 2275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:11.516736 2275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:11.518066 2275 hybrid_clock.cc:648] HybridClock initialized: now 1781874011517972 us; error 62 us; skew 500 ppm
I20260619 13:00:11.518661 2275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:11 dist-test-slave-tpsm krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:11.522213 2275 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:00:11.524097 2275 webserver.cc:460] Webserver started at http://127.1.87.67:36513/ using document root <none> and password file <none>
I20260619 13:00:11.524924 2275 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:11.525085 2275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:11.525420 2275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:11.529212 2275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "04a1b85fe26243d18b844e2742fb5224"
format_stamp: "Formatted at 2026-06-19 13:00:11 on dist-test-slave-tpsm"
server_key: "f58defc394aeb7e90137f039687cc2eb"
server_key_iv: "27cb4937308f00543583af09b0ff7671"
server_key_version: "encryptionkey@0"
I20260619 13:00:11.530239 2275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "04a1b85fe26243d18b844e2742fb5224"
format_stamp: "Formatted at 2026-06-19 13:00:11 on dist-test-slave-tpsm"
server_key: "f58defc394aeb7e90137f039687cc2eb"
server_key_iv: "27cb4937308f00543583af09b0ff7671"
server_key_version: "encryptionkey@0"
I20260619 13:00:11.535605 2275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:00:11.539453 2290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:11.541051 2275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:11.541246 2275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "04a1b85fe26243d18b844e2742fb5224"
format_stamp: "Formatted at 2026-06-19 13:00:11 on dist-test-slave-tpsm"
server_key: "f58defc394aeb7e90137f039687cc2eb"
server_key_iv: "27cb4937308f00543583af09b0ff7671"
server_key_version: "encryptionkey@0"
I20260619 13:00:11.541462 2275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:11.556241 2275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:11.560936 2275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:11.561291 2275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:11.562400 2275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:11.563948 2275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:11.564062 2275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:11.564222 2275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:11.564306 2275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:11.601531 2275 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:37887
I20260619 13:00:11.601616 2402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:37887 every 8 connection(s)
I20260619 13:00:11.602806 2275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:11.613219 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 2275
I20260619 13:00:11.613436 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:11.613897 1373 external_mini_cluster.cc:1428] Setting key dfa7c5e9be849dc32b1dda134256e8c1
Jun 19 13:00:11 dist-test-slave-tpsm krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:11.623104 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.605789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:56745 (local address 127.1.87.126:34747)
0619 13:00:11.605952 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.605956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.607191 (+ 1235us) server_negotiation.cc:407] Connection header received
0619 13:00:11.609104 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.609110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.609203 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.609336 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:11.610844 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.611676 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.613604 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.614056 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.614162 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.618136 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:11.618140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:11.618143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:11.618175 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:11.620596 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.621275 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.621279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.621281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.621395 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.621758 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.621761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.621763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.622006 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:11.622113 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.622704 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.622865 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:00:11.624100 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.606292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:34747 (local address 127.1.87.67:56745)
0619 13:00:11.607017 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:11.607066 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:11.608824 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:11.609573 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:11.609581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:11.610116 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:11.610669 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.610686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.611830 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.611842 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.613394 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.613413 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.614295 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.614303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.614591 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.615418 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:11.615463 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:11.617963 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:11.620724 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:11.620731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:11.620744 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:11.621168 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:11.621487 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:11.621490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:11.621492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:11.621659 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:11.622105 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:11.622110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:11.622412 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 13:00:11.622625 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 13:00:11.622982 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":287,"threads_started":1}
I20260619 13:00:11.625540 2403 heartbeater.cc:347] Connected to a master server at 127.1.87.126:34747
I20260619 13:00:11.625823 2403 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:11.626484 2403 heartbeater.cc:511] Master 127.1.87.126:34747 requested a full tablet report, sending...
I20260619 13:00:11.627961 1947 ts_manager.cc:194] Registered new tserver with Master: 04a1b85fe26243d18b844e2742fb5224 (127.1.87.67:37887)
I20260619 13:00:11.629033 1947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:56745
I20260619 13:00:11.629612 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:00:11.648558 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.635513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.1.87.126:34747)
0619 13:00:11.635673 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.635678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.636157 (+ 479us) server_negotiation.cc:407] Connection header received
0619 13:00:11.636364 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.636376 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.636434 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.636531 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:11.637950 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.638646 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.640408 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.640820 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.640949 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.643244 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:11.643254 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:11.643257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:11.643287 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:11.645817 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.646485 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.646492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.646496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.646632 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.646997 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.647003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.647007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.647376 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:11.647626 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.648022 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.648241 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:00:11.657642 2136 heartbeater.cc:503] Master 127.1.87.126:34747 was elected leader, sending a full tablet report...
I20260619 13:00:11.654734 1947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:00:11.661549 1947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:11.688357 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.674536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:37063 (local address 127.0.0.1:49136)
0619 13:00:11.675468 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:11.675513 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:11.675715 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:11.680823 (+ 5108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:11.680831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:11.680859 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:11.681484 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.681500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.682924 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.682932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.685312 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.685324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.687330 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.687336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.687577 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.687644 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 13:00:11.687755 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 13:00:11.687963 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":6,"thread_start_us":393,"threads_started":1}
I20260619 13:00:11.689430 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.674762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.1.87.66:37063)
0619 13:00:11.679977 (+ 5215us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.679987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.680023 (+ 36us) server_negotiation.cc:407] Connection header received
0619 13:00:11.680161 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.680172 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.680564 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.681581 (+ 1017us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:11.681801 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.682785 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.685493 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.687077 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.688627 (+ 1550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.688841 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.688912 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.689073 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4993,"thread_start_us":4882,"threads_started":1}
I20260619 13:00:11.690025 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.677060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.1.87.65:42995)
0619 13:00:11.678143 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.678156 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.680865 (+ 2709us) server_negotiation.cc:407] Connection header received
0619 13:00:11.681143 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.681151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.681366 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.681531 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:11.683061 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.684688 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.687029 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.688474 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.688865 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.689021 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.689548 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.689707 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":501,"threads_started":1}
I20260619 13:00:11.690070 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.675007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:37887 (local address 127.0.0.1:53752)
0619 13:00:11.676491 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:11.676513 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:11.676638 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:11.681130 (+ 4492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:11.681135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:11.681167 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:11.682643 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.682657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.685088 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.685093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.687559 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.687571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.689280 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.689286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.689443 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.689481 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:00:11.689559 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 13:00:11.689694 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"mutex_wait_us":1,"thread_start_us":5484,"threads_started":1}
I20260619 13:00:11.691175 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.675848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42995 (local address 127.0.0.1:47488)
0619 13:00:11.680758 (+ 4910us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:11.680774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:11.680959 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:11.682289 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:11.682298 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:11.682481 (+ 183us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:11.682871 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.682882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.684885 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.684891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.686856 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.686867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.689130 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.689134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.689345 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.689380 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:00:11.690280 (+ 900us) client_negotiation.cc:240] Negotiation successful
0619 13:00:11.690406 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4773,"mutex_wait_us":27,"thread_start_us":1922,"threads_started":1}
I20260619 13:00:11.691349 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.676296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.1.87.67:37887)
0619 13:00:11.680558 (+ 4262us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.680567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.680600 (+ 33us) server_negotiation.cc:407] Connection header received
0619 13:00:11.680697 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.680707 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.680921 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.682869 (+ 1948us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:11.683135 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.684922 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.687755 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.689081 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.689216 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.689433 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.690368 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.690519 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3970,"thread_start_us":3828,"threads_started":1}
I20260619 13:00:11.693204 2205 tablet_service.cc:1467] Processing CreateTablet for tablet 12b597ef696b4204adb2f75dc54c852f (DEFAULT_TABLE table=test-table [id=296291e1cd4c42b0ab149c6a5d004478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:11.694310 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b597ef696b4204adb2f75dc54c852f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:11.695281 2071 tablet_service.cc:1467] Processing CreateTablet for tablet 12b597ef696b4204adb2f75dc54c852f (DEFAULT_TABLE table=test-table [id=296291e1cd4c42b0ab149c6a5d004478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:11.695281 2338 tablet_service.cc:1467] Processing CreateTablet for tablet 12b597ef696b4204adb2f75dc54c852f (DEFAULT_TABLE table=test-table [id=296291e1cd4c42b0ab149c6a5d004478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:11.696640 2071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b597ef696b4204adb2f75dc54c852f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:11.696653 2338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b597ef696b4204adb2f75dc54c852f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:11.706456 2421 tablet_bootstrap.cc:492] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302: Bootstrap starting.
I20260619 13:00:11.710052 2421 tablet_bootstrap.cc:654] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:11.711272 2421 log.cc:824] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:11.712234 2423 tablet_bootstrap.cc:492] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7: Bootstrap starting.
I20260619 13:00:11.713177 2422 tablet_bootstrap.cc:492] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224: Bootstrap starting.
I20260619 13:00:11.715991 2421 tablet_bootstrap.cc:492] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302: No bootstrap required, opened a new log
I20260619 13:00:11.716277 2421 ts_tablet_manager.cc:1397] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 13:00:11.716593 2422 tablet_bootstrap.cc:654] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:11.716681 2423 tablet_bootstrap.cc:654] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:11.717593 2422 log.cc:824] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:11.718492 2423 log.cc:824] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:11.720430 2422 tablet_bootstrap.cc:492] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224: No bootstrap required, opened a new log
I20260619 13:00:11.720716 2422 ts_tablet_manager.cc:1397] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 13:00:11.723017 2423 tablet_bootstrap.cc:492] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7: No bootstrap required, opened a new log
I20260619 13:00:11.723278 2423 ts_tablet_manager.cc:1397] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 13:00:11.727545 2421 raft_consensus.cc:348] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } }
I20260619 13:00:11.728005 2421 raft_consensus.cc:374] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:11.728118 2421 raft_consensus.cc:729] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeaf88fc40db4542b3b3ca2650609302, State: Initialized, Role: FOLLOWER
I20260619 13:00:11.728132 2422 raft_consensus.cc:348] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } }
I20260619 13:00:11.728535 2422 raft_consensus.cc:374] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:11.728652 2422 raft_consensus.cc:729] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a1b85fe26243d18b844e2742fb5224, State: Initialized, Role: FOLLOWER
I20260619 13:00:11.728826 2421 consensus_queue.cc:260] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } }
I20260619 13:00:11.729244 2422 consensus_queue.cc:260] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } }
I20260619 13:00:11.730372 2270 heartbeater.cc:503] Master 127.1.87.126:34747 was elected leader, sending a full tablet report...
I20260619 13:00:11.730652 2421 ts_tablet_manager.cc:1428] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 13:00:11.731666 2403 heartbeater.cc:503] Master 127.1.87.126:34747 was elected leader, sending a full tablet report...
I20260619 13:00:11.732787 2422 ts_tablet_manager.cc:1428] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 13:00:11.735225 2423 raft_consensus.cc:348] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } }
I20260619 13:00:11.735710 2423 raft_consensus.cc:374] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:11.735821 2423 raft_consensus.cc:729] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185273f370fb45189c36c6fa466385d7, State: Initialized, Role: FOLLOWER
I20260619 13:00:11.736363 2423 consensus_queue.cc:260] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } }
I20260619 13:00:11.737982 2423 ts_tablet_manager.cc:1428] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260619 13:00:11.814101 2428 raft_consensus.cc:482] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:11.814414 2428 raft_consensus.cc:504] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } }
I20260619 13:00:11.816088 2428 leader_election.cc:288] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeaf88fc40db4542b3b3ca2650609302 (127.1.87.66:37063), 185273f370fb45189c36c6fa466385d7 (127.1.87.65:42995)
I20260619 13:00:11.822540 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.816598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:37063 (local address 127.1.87.67:57375)
0619 13:00:11.816815 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:11.816836 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:11.817002 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:11.817327 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:11.817331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:11.817358 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:11.817589 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.817603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.818631 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.818639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.820380 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.820393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.821522 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.821528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.822125 (+ 597us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.822170 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 13:00:11.822246 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 13:00:11.822342 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 13:00:11.822768 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.816600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:57375 (local address 127.1.87.66:37063)
0619 13:00:11.816818 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.816826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.816919 (+ 93us) server_negotiation.cc:407] Connection header received
0619 13:00:11.817135 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.817143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.817211 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.817310 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:11.817765 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.818475 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.820552 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.821351 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.821460 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.821530 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.822301 (+ 771us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.822510 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:00:11.823777 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b597ef696b4204adb2f75dc54c852f" candidate_uuid: "04a1b85fe26243d18b844e2742fb5224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeaf88fc40db4542b3b3ca2650609302" is_pre_election: true
I20260619 13:00:11.824162 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.817043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42995 (local address 127.1.87.67:38721)
0619 13:00:11.817883 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:11.817896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:11.818026 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:11.818332 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:11.818336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:11.818358 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:11.818779 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.818789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.820071 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.820077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.822014 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:11.822027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.823595 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.823600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:11.823719 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.823740 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 13:00:11.823814 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 13:00:11.823900 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":545,"threads_started":1}
I20260619 13:00:11.824244 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.817316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:38721 (local address 127.1.87.65:42995)
0619 13:00:11.817547 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.817556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.817984 (+ 428us) server_negotiation.cc:407] Connection header received
0619 13:00:11.818166 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.818175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.818229 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.818320 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:11.818943 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.819897 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.822177 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.823417 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.823562 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.823679 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.823869 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.824012 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:00:11.824491 2225 raft_consensus.cc:2454] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a1b85fe26243d18b844e2742fb5224 in term 0.
I20260619 13:00:11.825217 2291 leader_election.cc:302] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a1b85fe26243d18b844e2742fb5224, aeaf88fc40db4542b3b3ca2650609302; no voters:
I20260619 13:00:11.825611 2428 raft_consensus.cc:2781] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:11.825723 2428 raft_consensus.cc:482] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:11.825839 2428 raft_consensus.cc:3037] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:11.825865 2091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b597ef696b4204adb2f75dc54c852f" candidate_uuid: "04a1b85fe26243d18b844e2742fb5224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "185273f370fb45189c36c6fa466385d7" is_pre_election: true
I20260619 13:00:11.826368 2091 raft_consensus.cc:2454] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a1b85fe26243d18b844e2742fb5224 in term 0.
I20260619 13:00:11.828944 2428 raft_consensus.cc:504] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } }
I20260619 13:00:11.829802 2428 leader_election.cc:288] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [CANDIDATE]: Term 1 election: Requested vote from peers aeaf88fc40db4542b3b3ca2650609302 (127.1.87.66:37063), 185273f370fb45189c36c6fa466385d7 (127.1.87.65:42995)
I20260619 13:00:11.830299 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b597ef696b4204adb2f75dc54c852f" candidate_uuid: "04a1b85fe26243d18b844e2742fb5224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeaf88fc40db4542b3b3ca2650609302"
I20260619 13:00:11.830547 2225 raft_consensus.cc:3037] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:11.830677 2091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b597ef696b4204adb2f75dc54c852f" candidate_uuid: "04a1b85fe26243d18b844e2742fb5224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "185273f370fb45189c36c6fa466385d7"
I20260619 13:00:11.830935 2091 raft_consensus.cc:3037] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:11.833595 2225 raft_consensus.cc:2454] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a1b85fe26243d18b844e2742fb5224 in term 1.
I20260619 13:00:11.834002 2291 leader_election.cc:302] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a1b85fe26243d18b844e2742fb5224, aeaf88fc40db4542b3b3ca2650609302; no voters:
I20260619 13:00:11.834188 2091 raft_consensus.cc:2454] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a1b85fe26243d18b844e2742fb5224 in term 1.
I20260619 13:00:11.834363 2428 raft_consensus.cc:2781] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:11.834823 2428 raft_consensus.cc:686] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [term 1 LEADER]: Becoming Leader. State: Replica: 04a1b85fe26243d18b844e2742fb5224, State: Running, Role: LEADER
I20260619 13:00:11.835456 2428 consensus_queue.cc:237] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } }
I20260619 13:00:11.840349 1946 catalog_manager.cc:5495] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04a1b85fe26243d18b844e2742fb5224 (127.1.87.67). New cstate: current_term: 1 leader_uuid: "04a1b85fe26243d18b844e2742fb5224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeaf88fc40db4542b3b3ca2650609302" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185273f370fb45189c36c6fa466385d7" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a1b85fe26243d18b844e2742fb5224" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37887 } health_report { overall_health: HEALTHY } } }
W20260619 13:00:11.854998 2404 tablet.cc:2369] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:00:11.860085 2271 tablet.cc:2369] T 12b597ef696b4204adb2f75dc54c852f P aeaf88fc40db4542b3b3ca2650609302: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:00:11.878603 2137 tablet.cc:2369] T 12b597ef696b4204adb2f75dc54c852f P 185273f370fb45189c36c6fa466385d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 13:00:11 dist-test-slave-tpsm krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:11.888180 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.872631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.1.87.65:42995)
0619 13:00:11.872802 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.872807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.873318 (+ 511us) server_negotiation.cc:407] Connection header received
0619 13:00:11.873535 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.873544 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.873608 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.873722 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:11.875226 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.876088 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.877352 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.877796 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.877959 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.882576 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:11.882586 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:11.882601 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:11.882635 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:11.885449 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.886112 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.886128 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.886131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.886279 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.886639 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.886646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.886649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.886999 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:11.887346 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.887604 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.887813 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 13:00:11 dist-test-slave-tpsm krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:00:11 dist-test-slave-tpsm krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:11.939963 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.921264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.1.87.65:42995)
0619 13:00:11.921645 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.921652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.921681 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:00:11.921869 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.921875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.921948 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.922055 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:11.923595 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.924429 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.926013 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.926546 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.926660 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.934667 (+ 8007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:11.934673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:11.934676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:11.934717 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:11.937156 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.937861 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.937866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.937868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.938040 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.938552 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.938556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.938558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.938862 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:11.939056 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.939428 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.939603 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
Jun 19 13:00:11 dist-test-slave-tpsm krb5kdc[1902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:00:11 dist-test-slave-tpsm krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:11.983130 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.970106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.1.87.65:42995)
0619 13:00:11.970378 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.970387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.970544 (+ 157us) server_negotiation.cc:407] Connection header received
0619 13:00:11.970694 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.970698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.970761 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.970847 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:11.972184 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.972860 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.974006 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.974445 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.974557 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.978291 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:11.978296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:11.978299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:11.978331 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:11.980734 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.981284 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.981288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.981290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.981489 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:11.981841 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:11.981844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:11.981846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:11.982108 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:11.982287 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:11.982581 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 13:00:11.982758 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260619 13:00:11.986388 2051 tablet_service.cc:2908] Rejecting scan request for tablet 12b597ef696b4204adb2f75dc54c852f: Uninitialized: clean time has not yet been initialized
I20260619 13:00:11.989930 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2008
I20260619 13:00:12.002054 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:11.990832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.1.87.65:42995)
0619 13:00:11.991067 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:11.991073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:11.991370 (+ 297us) server_negotiation.cc:407] Connection header received
0619 13:00:11.991517 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:11.991521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:11.991596 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:11.991673 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:11.992734 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.993393 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.994513 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:11.994920 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:11.995019 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:11.997083 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:11.997088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:11.997092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:11.997123 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:11.999807 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:12.000380 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:12.000384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:12.000387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:12.000519 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:12.000811 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:12.000814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:12.000816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:12.001097 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:12.001310 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:12.001488 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 13:00:12.001658 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:00:12.002764 2131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:12.104516 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 2008
I20260619 13:00:12.128522 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2142
Jun 19 13:00:12 dist-test-slave-tpsm krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:00:12.143131 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:12.129269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.1.87.66:37063)
0619 13:00:12.129432 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:12.129439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:12.129526 (+ 87us) server_negotiation.cc:407] Connection header received
0619 13:00:12.129734 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:12.129738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:12.129797 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:12.129901 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:12.131079 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:12.131915 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:12.133172 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:12.133546 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:12.133677 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:12.137397 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:12.137412 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:12.137427 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:12.137461 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:12.140135 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:12.140765 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:12.140772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:12.140775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:12.140890 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:12.141262 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:12.141268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:12.141273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:12.141608 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:12.141909 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:12.142165 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 13:00:12.142617 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:00:12.143801 2265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:12.222370 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:12.221932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42995 (local address 127.1.87.67:48887)
0619 13:00:12.222118 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:12.222221 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 13:00:12.222633 2291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.87.65:42995: connect: Connection refused (error 111)
I20260619 13:00:12.224103 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:12.223760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42995 (local address 127.1.87.67:59443)
0619 13:00:12.223910 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:12.223983 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:00:12.224290 2291 consensus_peers.cc:487] T 12b597ef696b4204adb2f75dc54c852f P 04a1b85fe26243d18b844e2742fb5224 -> Peer 185273f370fb45189c36c6fa466385d7 (127.1.87.65:42995): Couldn't send request to peer 185273f370fb45189c36c6fa466385d7. Status: Network error: Client connection negotiation failed: client connection to 127.1.87.65:42995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 13:00:12.232844 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 2142
I20260619 13:00:12.250746 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2275
Jun 19 13:00:12 dist-test-slave-tpsm krb5kdc[1902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:00:12.270823 2447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:12.251762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.1.87.67:37887)
0619 13:00:12.252291 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:12.252295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:12.252311 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:00:12.252362 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:12.252365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:12.252422 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:12.252519 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:12.253767 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:12.254443 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:12.255908 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:12.256314 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:12.256420 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:12.264795 (+ 8375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:12.264811 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:12.264826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:12.264864 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:12.267776 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:12.268529 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:12.268540 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:12.268543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:12.268780 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:12.269215 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:12.269221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:12.269224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:12.269725 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:12.269991 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:12.270187 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 13:00:12.270366 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":227,"threads_started":1}
I20260619 13:00:12.271589 2398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:12.361194 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 2275
I20260619 13:00:12.378130 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 1915
I20260619 13:00:12.379405 1977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:12.490337 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 1915
2026-06-19T13:00:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4123 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:12 dist-test-slave-tpsm krb5kdc[2452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:12 dist-test-slave-tpsm krb5kdc[2452](info): set up 2 sockets
Jun 19 13:00:12 dist-test-slave-tpsm krb5kdc[2452](info): commencing operation
krb5kdc: starting...
W20260619 13:00:13.567302 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:13 dist-test-slave-tpsm krb5kdc[2452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:13.740404 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:42115
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:14.052642 2465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:14.052991 2465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:14.053153 2465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:14.060279 2465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:14.060434 2465 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 13:00:14.060529 2465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:14.060649 2465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:14.060740 2465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:14.060832 2465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:14.070830 2465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:42115
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:14.072202 2465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:14.073527 2465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:14.085561 2470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:14.085563 2473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:14.085628 2471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:14.086954 2465 server_base.cc:1034] running on GCE node
I20260619 13:00:14.087777 2465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:14.089161 2465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:14.090458 2465 hybrid_clock.cc:648] HybridClock initialized: now 1781874014090390 us; error 36 us; skew 500 ppm
I20260619 13:00:14.090817 2465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:14 dist-test-slave-tpsm krb5kdc[2452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:14.094094 2465 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:14.095659 2465 webserver.cc:460] Webserver started at http://127.1.87.126:32913/ using document root <none> and password file <none>
I20260619 13:00:14.096369 2465 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:14.096500 2465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:14.096956 2465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:14.099895 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "4e7b3c0182484fbea33933ef878372d2"
format_stamp: "Formatted at 2026-06-19 13:00:14 on dist-test-slave-tpsm"
server_key: "8baffc1d4f2221569b80fe64f900b9cb"
server_key_iv: "b4fffcf484eeb23685616ed31fd363e8"
server_key_version: "encryptionkey@0"
I20260619 13:00:14.100657 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "4e7b3c0182484fbea33933ef878372d2"
format_stamp: "Formatted at 2026-06-19 13:00:14 on dist-test-slave-tpsm"
server_key: "8baffc1d4f2221569b80fe64f900b9cb"
server_key_iv: "b4fffcf484eeb23685616ed31fd363e8"
server_key_version: "encryptionkey@0"
I20260619 13:00:14.105847 2465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:00:14.109264 2480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:14.110683 2465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:14.110872 2465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "4e7b3c0182484fbea33933ef878372d2"
format_stamp: "Formatted at 2026-06-19 13:00:14 on dist-test-slave-tpsm"
server_key: "8baffc1d4f2221569b80fe64f900b9cb"
server_key_iv: "b4fffcf484eeb23685616ed31fd363e8"
server_key_version: "encryptionkey@0"
I20260619 13:00:14.111101 2465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:14.139948 2465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:14.143939 2465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:14.144233 2465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:14.166285 2465 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:42115
I20260619 13:00:14.166306 2541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:42115 every 8 connection(s)
I20260619 13:00:14.167774 2465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:14.171747 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:14.177009 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 2465
I20260619 13:00:14.177242 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:14.178328 1373 external_mini_cluster.cc:1428] Setting key a185d63765080b7cb1aad44ed32a93e1
I20260619 13:00:14.182665 2542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2: Bootstrap starting.
I20260619 13:00:14.186264 2542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:14.187295 2542 log.cc:824] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:14.190124 2542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2: No bootstrap required, opened a new log
Jun 19 13:00:14 dist-test-slave-tpsm krb5kdc[2452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:14.198694 2542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b3c0182484fbea33933ef878372d2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42115 } }
I20260619 13:00:14.199079 2542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:14.199205 2542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7b3c0182484fbea33933ef878372d2, State: Initialized, Role: FOLLOWER
I20260619 13:00:14.199856 2542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b3c0182484fbea33933ef878372d2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42115 } }
I20260619 13:00:14.200098 2542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:14.200249 2542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:14.200428 2542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:14.201742 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:14.180829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.1.87.126:42115)
0619 13:00:14.181782 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:14.181794 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:14.181833 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:00:14.183608 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:14.183637 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:14.184009 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:14.184327 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:14.186072 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.187638 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.189188 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.189831 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.189984 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:14.194485 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:14.194492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:14.194507 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:14.194545 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:14.197638 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:14.198307 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:14.198315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:14.198330 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:14.198483 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:14.198846 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:14.198852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:14.198855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:14.199542 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:14.199846 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:14.200271 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 13:00:14.200670 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":496,"threads_started":1}
I20260619 13:00:14.203370 2542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b3c0182484fbea33933ef878372d2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42115 } }
I20260619 13:00:14.203918 2542 leader_election.cc:302] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e7b3c0182484fbea33933ef878372d2; no voters:
I20260619 13:00:14.204643 2542 leader_election.cc:288] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:14.204821 2547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:14.205606 2547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e7b3c0182484fbea33933ef878372d2, State: Running, Role: LEADER
I20260619 13:00:14.206171 2547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b3c0182484fbea33933ef878372d2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42115 } }
I20260619 13:00:14.206723 2542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:14.209787 2548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e7b3c0182484fbea33933ef878372d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b3c0182484fbea33933ef878372d2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42115 } } }
I20260619 13:00:14.210173 2548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:14.209762 2549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e7b3c0182484fbea33933ef878372d2. Latest consensus state: current_term: 1 leader_uuid: "4e7b3c0182484fbea33933ef878372d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e7b3c0182484fbea33933ef878372d2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42115 } } }
I20260619 13:00:14.210597 2549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:14.215764 2555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:14.226337 2555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:14.235862 2555 catalog_manager.cc:1347] Generated new cluster ID: 160e287a6d46499e9e7dce915460ef4c
I20260619 13:00:14.236004 2555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:14.268443 2555 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:14.269690 2555 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:14.274658 2555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2: Generated new TSK 0
I20260619 13:00:14.275625 2555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:00:14.328747 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:42115
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:14.656381 2568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:14.656731 2568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:14.656903 2568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:14.664446 2568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:14.664634 2568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:14.664814 2568 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:00:14.664963 2568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:14.675014 2568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.87.126:42115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:14.676406 2568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:14.677820 2568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:14.690558 2576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:14.690622 2573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:14.690600 2574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:14.691877 2568 server_base.cc:1034] running on GCE node
I20260619 13:00:14.692518 2568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:14.693468 2568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:14.694737 2568 hybrid_clock.cc:648] HybridClock initialized: now 1781874014694659 us; error 51 us; skew 500 ppm
I20260619 13:00:14.695065 2568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:14 dist-test-slave-tpsm krb5kdc[2452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:14.698304 2568 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:00:14.699927 2568 webserver.cc:460] Webserver started at http://127.1.87.65:40715/ using document root <none> and password file <none>
I20260619 13:00:14.700700 2568 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:14.700838 2568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:14.701190 2568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:14.704344 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "7d483c83583b41bd8b70a84e5170fda4"
format_stamp: "Formatted at 2026-06-19 13:00:14 on dist-test-slave-tpsm"
server_key: "f9fdf4d60130f06c49af1c4ec6fd6fb5"
server_key_iv: "8924d72adc2bb6bd7075725b4629f48b"
server_key_version: "encryptionkey@0"
I20260619 13:00:14.705189 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "7d483c83583b41bd8b70a84e5170fda4"
format_stamp: "Formatted at 2026-06-19 13:00:14 on dist-test-slave-tpsm"
server_key: "f9fdf4d60130f06c49af1c4ec6fd6fb5"
server_key_iv: "8924d72adc2bb6bd7075725b4629f48b"
server_key_version: "encryptionkey@0"
I20260619 13:00:14.710737 2568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 13:00:14.714428 2583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:14.715965 2568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 13:00:14.716125 2568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "7d483c83583b41bd8b70a84e5170fda4"
format_stamp: "Formatted at 2026-06-19 13:00:14 on dist-test-slave-tpsm"
server_key: "f9fdf4d60130f06c49af1c4ec6fd6fb5"
server_key_iv: "8924d72adc2bb6bd7075725b4629f48b"
server_key_version: "encryptionkey@0"
I20260619 13:00:14.716354 2568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:14.735289 2568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:14.739756 2568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:14.740041 2568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:14.742969 2568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:14.743093 2568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:14.743263 2568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:14.743371 2568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:00:14 dist-test-slave-tpsm krb5kdc[2452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:14.767959 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:14.743768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:45779 (local address 127.1.87.126:42115)
0619 13:00:14.744337 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:14.744345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:14.745565 (+ 1220us) server_negotiation.cc:407] Connection header received
0619 13:00:14.748391 (+ 2826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:14.748397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:14.748492 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:14.748617 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:14.751635 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.752641 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.755785 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.756310 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.756454 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:14.761121 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:14.761126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:14.761129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:14.761162 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:14.763777 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:14.764798 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:14.764805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:14.764812 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:14.764986 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:14.765517 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:14.765523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:14.765526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:14.765919 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:14.766060 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:14.767077 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0619 13:00:14.767414 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":349,"threads_started":1}
I20260619 13:00:14.768626 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:14.744446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42115 (local address 127.1.87.65:45779)
0619 13:00:14.745361 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:14.745415 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:14.747933 (+ 2518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:14.748991 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:14.749020 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:14.750135 (+ 1115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:14.751360 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:14.751399 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.752853 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.752860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:14.755603 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:14.755626 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.756604 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.756614 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:14.756975 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:14.757914 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:14.757955 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:14.760876 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:14.763933 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:14.763943 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:14.763962 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:14.764591 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:14.765124 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:14.765135 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:14.765138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:14.765394 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:14.766054 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:14.766062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:14.766557 (+ 495us) client_negotiation.cc:769] Sending connection context
0619 13:00:14.766924 (+ 367us) client_negotiation.cc:240] Negotiation successful
0619 13:00:14.767409 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":383,"threads_started":1}
I20260619 13:00:14.788951 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:14.774144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:44275 (local address 127.1.87.126:42115)
0619 13:00:14.774370 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:14.774385 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:14.775969 (+ 1584us) server_negotiation.cc:407] Connection header received
0619 13:00:14.776088 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:14.776096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:14.776198 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:14.776332 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:14.777204 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.777893 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.779136 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.779597 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.779737 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:14.782458 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:14.782465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:14.782471 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:14.782512 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:14.786075 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:14.787048 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:14.787052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:14.787054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:14.787180 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:14.787612 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:14.787615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:14.787617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:14.787861 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:14.788010 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:14.788399 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 13:00:14.788606 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:00:14.789152 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:14.774129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42115 (local address 127.1.87.65:44275)
0619 13:00:14.775870 (+ 1741us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:14.775885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:14.775991 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:14.776342 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:14.776345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:14.776764 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:14.777050 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:14.777058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.778018 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.778026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:14.778996 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:14.779006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:14.779777 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:14.779784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:14.780013 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:14.780836 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:14.780872 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:14.782259 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:14.786262 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:14.786268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:14.786272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:14.786924 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:14.787299 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:14.787304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:14.787308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:14.787509 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:14.787979 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:14.787984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:14.788157 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 13:00:14.788311 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 13:00:14.788653 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1583,"thread_start_us":1400,"threads_started":1}
I20260619 13:00:14.798962 2568 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:45399
I20260619 13:00:14.798980 2703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:45399 every 8 connection(s)
I20260619 13:00:14.800236 2568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:14.802167 2704 heartbeater.cc:347] Connected to a master server at 127.1.87.126:42115
I20260619 13:00:14.802350 2704 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:14.802898 2704 heartbeater.cc:511] Master 127.1.87.126:42115 requested a full tablet report, sending...
I20260619 13:00:14.804791 2497 ts_manager.cc:194] Registered new tserver with Master: 7d483c83583b41bd8b70a84e5170fda4 (127.1.87.65:45399)
I20260619 13:00:14.805739 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 2568
I20260619 13:00:14.805958 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:14.806295 2497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:45779
I20260619 13:00:14.806640 1373 external_mini_cluster.cc:1428] Setting key d3d7defc2b1ada4663853664ecd7459f
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:00:14.844417 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:42115
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:15.178741 2708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:15.179056 2708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:15.179239 2708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:15.186729 2708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:15.186852 2708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:15.187005 2708 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:00:15.187148 2708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:15.197855 2708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.87.126:42115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:15.199177 2708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:15.200644 2708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:15.214041 2716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:15.214020 2713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:15.214259 2714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:15.221156 2708 server_base.cc:1034] running on GCE node
I20260619 13:00:15.221834 2708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:15.222806 2708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:15.224076 2708 hybrid_clock.cc:648] HybridClock initialized: now 1781874015224006 us; error 44 us; skew 500 ppm
I20260619 13:00:15.224413 2708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:15 dist-test-slave-tpsm krb5kdc[2452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874015, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:15.227644 2708 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:00:15.229357 2708 webserver.cc:460] Webserver started at http://127.1.87.66:38945/ using document root <none> and password file <none>
I20260619 13:00:15.230094 2708 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:15.230219 2708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:15.230557 2708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:15.233486 2708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "0bb196b5a55243ab99f82ce7a8a8197e"
format_stamp: "Formatted at 2026-06-19 13:00:15 on dist-test-slave-tpsm"
server_key: "3f0368b0740f95346a159730a278d5c0"
server_key_iv: "7deaf8d91f2d6e4ea44d120030db25a2"
server_key_version: "encryptionkey@0"
I20260619 13:00:15.234326 2708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "0bb196b5a55243ab99f82ce7a8a8197e"
format_stamp: "Formatted at 2026-06-19 13:00:15 on dist-test-slave-tpsm"
server_key: "3f0368b0740f95346a159730a278d5c0"
server_key_iv: "7deaf8d91f2d6e4ea44d120030db25a2"
server_key_version: "encryptionkey@0"
I20260619 13:00:15.239821 2708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:00:15.243436 2723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:15.244812 2708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:00:15.245014 2708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "0bb196b5a55243ab99f82ce7a8a8197e"
format_stamp: "Formatted at 2026-06-19 13:00:15 on dist-test-slave-tpsm"
server_key: "3f0368b0740f95346a159730a278d5c0"
server_key_iv: "7deaf8d91f2d6e4ea44d120030db25a2"
server_key_version: "encryptionkey@0"
I20260619 13:00:15.245293 2708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:15.261899 2708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:15.267225 2708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:15.267529 2708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:15.271564 2708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:15.271685 2708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:15.271832 2708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:15.271932 2708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:00:15 dist-test-slave-tpsm krb5kdc[2452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874015, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:15.295830 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.271560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:44165 (local address 127.1.87.126:42115)
0619 13:00:15.271773 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:15.271780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:15.273432 (+ 1652us) server_negotiation.cc:407] Connection header received
0619 13:00:15.275671 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:15.275678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:15.275772 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:15.275910 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:15.278170 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.278895 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.281611 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.282577 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.282752 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.287055 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:15.287065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:15.287074 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:15.287130 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:15.291552 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.292588 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.292598 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.292604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.292780 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.293572 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.293581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.293588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.293997 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:15.294184 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:15.295361 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0619 13:00:15.295586 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:00:15.297890 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.272236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42115 (local address 127.1.87.66:44165)
0619 13:00:15.273222 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:15.273273 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:15.275384 (+ 2111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:15.276273 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:15.276284 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:15.277127 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:15.277971 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.277993 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.279057 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.279068 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.281435 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.281452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.282812 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.282817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.283001 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.283832 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:15.283867 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:15.286793 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:15.291767 (+ 4974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:15.291784 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:15.291811 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:15.292449 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:15.293053 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:15.293061 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:15.293067 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:15.293410 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:15.294172 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:15.294181 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:15.294781 (+ 600us) client_negotiation.cc:769] Sending connection context
0619 13:00:15.295215 (+ 434us) client_negotiation.cc:240] Negotiation successful
0619 13:00:15.295951 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":451,"threads_started":1}
I20260619 13:00:15.320056 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.304046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:59377 (local address 127.1.87.126:42115)
0619 13:00:15.304244 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:15.304252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:15.305132 (+ 880us) server_negotiation.cc:407] Connection header received
0619 13:00:15.305338 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:15.305341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:15.305403 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:15.305521 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:15.306437 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.307254 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.308942 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.309696 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.309847 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.312681 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:15.312689 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:15.312697 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:15.312739 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:15.316748 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.317746 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.317749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.317752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.317866 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.318275 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.318278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.318280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.318633 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:15.318794 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:15.319387 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 13:00:15.319658 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:00:15.320670 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.304120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42115 (local address 127.1.87.66:59377)
0619 13:00:15.305003 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:15.305028 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:15.305215 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:15.305560 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:15.305570 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:15.306023 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:15.306297 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.306306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.307413 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.307419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.308804 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.308815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.309923 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.309933 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.310232 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.311010 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:15.311036 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:15.312466 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:15.316898 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:15.316905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:15.316911 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:15.317622 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:15.317986 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:15.317990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:15.317994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:15.318159 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:15.318772 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:15.318780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:15.319044 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 13:00:15.319248 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 13:00:15.319855 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":533,"threads_started":1}
I20260619 13:00:15.326721 2708 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:36919
I20260619 13:00:15.326745 2842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:36919 every 8 connection(s)
I20260619 13:00:15.327901 2708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:15.330112 2843 heartbeater.cc:347] Connected to a master server at 127.1.87.126:42115
I20260619 13:00:15.330328 2843 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:15.330933 2843 heartbeater.cc:511] Master 127.1.87.126:42115 requested a full tablet report, sending...
I20260619 13:00:15.332207 2497 ts_manager.cc:194] Registered new tserver with Master: 0bb196b5a55243ab99f82ce7a8a8197e (127.1.87.66:36919)
I20260619 13:00:15.332930 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 2708
I20260619 13:00:15.333107 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:15.333290 2497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:44165
I20260619 13:00:15.333664 1373 external_mini_cluster.cc:1428] Setting key 1529429a5e25bf1e403fbd1a8852ffea
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:00:15.370838 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:42115
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:15.690913 2847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:15.691179 2847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:15.691406 2847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:15.698419 2847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:15.698518 2847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:15.698674 2847 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:00:15.698791 2847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:15.708667 2847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.87.126:42115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:15.709894 2847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:15.711184 2847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:15.724095 2855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:15.724165 2852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:15.724165 2853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:15.725219 2847 server_base.cc:1034] running on GCE node
I20260619 13:00:15.725795 2847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:15.726722 2847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:15.727921 2847 hybrid_clock.cc:648] HybridClock initialized: now 1781874015727844 us; error 57 us; skew 500 ppm
I20260619 13:00:15.728209 2847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:15 dist-test-slave-tpsm krb5kdc[2452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874015, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:15.731299 2847 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:00:15.732838 2847 webserver.cc:460] Webserver started at http://127.1.87.67:45245/ using document root <none> and password file <none>
I20260619 13:00:15.733533 2847 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:15.733618 2847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:15.733868 2847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:15.736792 2847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "859c3f3212a242ecb661b3e5ad5b5e3c"
format_stamp: "Formatted at 2026-06-19 13:00:15 on dist-test-slave-tpsm"
server_key: "e55226a947f1f43e6b46bf4824939393"
server_key_iv: "86e26e057bbfea8a033e9cfee5d1700f"
server_key_version: "encryptionkey@0"
I20260619 13:00:15.737524 2847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "859c3f3212a242ecb661b3e5ad5b5e3c"
format_stamp: "Formatted at 2026-06-19 13:00:15 on dist-test-slave-tpsm"
server_key: "e55226a947f1f43e6b46bf4824939393"
server_key_iv: "86e26e057bbfea8a033e9cfee5d1700f"
server_key_version: "encryptionkey@0"
I20260619 13:00:15.742815 2847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:00:15.746168 2862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:15.747578 2847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:00:15.747731 2847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "859c3f3212a242ecb661b3e5ad5b5e3c"
format_stamp: "Formatted at 2026-06-19 13:00:15 on dist-test-slave-tpsm"
server_key: "e55226a947f1f43e6b46bf4824939393"
server_key_iv: "86e26e057bbfea8a033e9cfee5d1700f"
server_key_version: "encryptionkey@0"
I20260619 13:00:15.747946 2847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:15.769424 2847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:15.773787 2847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:15.774124 2847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:15.776973 2847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:15.777086 2847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:15.777208 2847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:15.777300 2847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:00:15 dist-test-slave-tpsm krb5kdc[2452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874015, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:15.800788 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.778182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42115 (local address 127.1.87.67:47489)
0619 13:00:15.778950 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:15.778986 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:15.780941 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:15.781739 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:15.781755 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:15.782555 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:15.783453 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.783471 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.784669 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.784678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.786995 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.787005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.788045 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.788055 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.788611 (+ 556us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.789598 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:15.789660 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:15.793225 (+ 3565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:15.796559 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:15.796574 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:15.796594 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:15.797232 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:15.797631 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:15.797635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:15.797639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:15.797830 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:15.798538 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:15.798547 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:15.798951 (+ 404us) client_negotiation.cc:769] Sending connection context
0619 13:00:15.799160 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 13:00:15.799634 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":345,"threads_started":1}
I20260619 13:00:15.801812 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.777527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:47489 (local address 127.1.87.126:42115)
0619 13:00:15.777740 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:15.777748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:15.779103 (+ 1355us) server_negotiation.cc:407] Connection header received
0619 13:00:15.781232 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:15.781237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:15.781326 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:15.781436 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:15.783623 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.784506 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.787137 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.787731 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.787898 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.793456 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:15.793461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:15.793464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:15.793498 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:15.796372 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.797367 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.797372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.797376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.797517 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.797951 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.797956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.797959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.798382 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:15.801263 (+ 2881us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:15.801407 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 13:00:15.801584 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:00:15.809244 2704 heartbeater.cc:503] Master 127.1.87.126:42115 was elected leader, sending a full tablet report...
I20260619 13:00:15.824993 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.810244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:45441 (local address 127.1.87.126:42115)
0619 13:00:15.810414 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:15.810419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:15.810437 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:00:15.810637 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:15.810652 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:15.810725 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:15.810790 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:15.812161 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.812874 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.814821 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.815564 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.815716 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.818282 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:15.818287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:15.818290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:15.818318 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:15.821753 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.822890 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.822895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.822899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.823023 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.823488 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.823495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.823500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.823818 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:15.823971 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:15.824409 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 13:00:15.824623 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:00:15.825223 2925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.809683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42115 (local address 127.1.87.67:45441)
0619 13:00:15.810341 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:15.810360 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:15.810485 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:15.810935 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:15.810938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:15.811447 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:15.812004 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.812015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.813004 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.813011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.814686 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.814698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.815789 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.815796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.815975 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.816660 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:15.816692 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:15.818071 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:15.821952 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:15.821957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:15.821961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:15.822733 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:15.823140 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:15.823148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:15.823155 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:15.823363 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:15.823941 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:15.823945 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:15.824135 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 13:00:15.824319 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 13:00:15.824689 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":746,"threads_started":1}
I20260619 13:00:15.835852 2847 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:40883
I20260619 13:00:15.835870 2981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:40883 every 8 connection(s)
I20260619 13:00:15.837046 2847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:15.839002 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 2847
I20260619 13:00:15.839192 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:15.839228 2982 heartbeater.cc:347] Connected to a master server at 127.1.87.126:42115
I20260619 13:00:15.839586 2982 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:15.839815 1373 external_mini_cluster.cc:1428] Setting key cf780c836ddbde14416c95620eb9b9b9
I20260619 13:00:15.840368 2982 heartbeater.cc:511] Master 127.1.87.126:42115 requested a full tablet report, sending...
I20260619 13:00:15.841804 2497 ts_manager.cc:194] Registered new tserver with Master: 859c3f3212a242ecb661b3e5ad5b5e3c (127.1.87.67:40883)
I20260619 13:00:15.842880 2497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:47489
I20260619 13:00:15.843168 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:00:15 dist-test-slave-tpsm krb5kdc[2452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:00:15 dist-test-slave-tpsm krb5kdc[2452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:15.889366 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.875272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.1.87.126:42115)
0619 13:00:15.875549 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:15.875570 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:15.875726 (+ 156us) server_negotiation.cc:407] Connection header received
0619 13:00:15.875946 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:15.875951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:15.876013 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:15.876124 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:15.877564 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.878289 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.879661 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.880071 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.880213 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.884121 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:15.884126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:15.884130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:15.884163 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:15.886679 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.887402 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.887406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.887409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.887531 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:15.887941 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:15.887950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:15.887952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:15.888215 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:15.888458 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:15.888771 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 13:00:15.888986 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260619 13:00:15.893852 2497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:00:15.900732 2497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:15.934362 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.914621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.1.87.66:36919)
0619 13:00:15.926373 (+ 11752us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:15.926381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:15.926412 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:00:15.926517 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:15.926527 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:15.926794 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:15.926992 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:15.927898 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.929419 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.931807 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.933223 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.933360 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.933586 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:15.933888 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 13:00:15.934069 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11590,"thread_start_us":11607,"threads_started":1}
I20260619 13:00:15.934410 2995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.913663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:36919 (local address 127.0.0.1:45036)
0619 13:00:15.925619 (+ 11956us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:15.925670 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:15.925889 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:15.927168 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:15.927176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:15.927216 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:15.927585 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.927598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.929550 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.929556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.931665 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.931679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.933433 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.933439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.933632 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.933684 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 13:00:15.933806 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 13:00:15.933999 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11609,"mutex_wait_us":3,"thread_start_us":11507,"threads_started":1}
I20260619 13:00:15.935838 2996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.913682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:40883 (local address 127.0.0.1:44014)
0619 13:00:15.926898 (+ 13216us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:15.926917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:15.927029 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:15.928053 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:15.928057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:15.928083 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:15.928540 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.928552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.930507 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.930513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.932854 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.932865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.935073 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.935078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.935180 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.935201 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 13:00:15.935309 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:00:15.935457 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13066,"mutex_wait_us":30,"thread_start_us":12851,"threads_started":1}
I20260619 13:00:15.935855 2997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.913949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.1.87.67:40883)
0619 13:00:15.925123 (+ 11174us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:15.925137 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:15.927146 (+ 2009us) server_negotiation.cc:407] Connection header received
0619 13:00:15.927354 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:15.927365 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:15.927589 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:15.927771 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:15.928754 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.930371 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.933005 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.934657 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.934780 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.934964 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:15.935382 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 13:00:15.935542 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10929,"thread_start_us":10683,"threads_started":1}
I20260619 13:00:15.936275 2999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.927286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.1.87.65:45399)
0619 13:00:15.927859 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:15.927868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:15.928612 (+ 744us) server_negotiation.cc:407] Connection header received
0619 13:00:15.928808 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:15.928815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:15.928968 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:15.929145 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:15.929717 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.930925 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.933557 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.934965 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.935057 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.935222 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:15.935830 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 13:00:15.935989 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":298,"threads_started":1}
I20260619 13:00:15.936983 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:15.927901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:45399 (local address 127.0.0.1:48210)
0619 13:00:15.928489 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:15.928505 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:15.928652 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:15.929135 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:15.929139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:15.929163 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:15.929512 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.929533 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.931063 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.931069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.933383 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:15.933396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:15.935423 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:15.935433 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:15.935610 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:15.935644 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:00:15.936514 (+ 870us) client_negotiation.cc:240] Negotiation successful
0619 13:00:15.936645 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":37,"thread_start_us":333,"threads_started":1}
I20260619 13:00:15.938423 2773 tablet_service.cc:1467] Processing CreateTablet for tablet 130d714f1584404086ca68006968c92e (DEFAULT_TABLE table=test-table [id=b4cfe1ea88e34b09a17175314bbf2950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:15.939766 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130d714f1584404086ca68006968c92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:15.939275 2912 tablet_service.cc:1467] Processing CreateTablet for tablet 130d714f1584404086ca68006968c92e (DEFAULT_TABLE table=test-table [id=b4cfe1ea88e34b09a17175314bbf2950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:15.940322 2634 tablet_service.cc:1467] Processing CreateTablet for tablet 130d714f1584404086ca68006968c92e (DEFAULT_TABLE table=test-table [id=b4cfe1ea88e34b09a17175314bbf2950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:15.940893 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130d714f1584404086ca68006968c92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:15.941560 2634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130d714f1584404086ca68006968c92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:15.950872 3001 tablet_bootstrap.cc:492] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: Bootstrap starting.
I20260619 13:00:15.954151 3001 tablet_bootstrap.cc:654] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:15.955605 3002 tablet_bootstrap.cc:492] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: Bootstrap starting.
I20260619 13:00:15.957021 3001 log.cc:824] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:15.958428 3003 tablet_bootstrap.cc:492] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: Bootstrap starting.
I20260619 13:00:15.959276 3002 tablet_bootstrap.cc:654] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:15.960083 3001 tablet_bootstrap.cc:492] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: No bootstrap required, opened a new log
I20260619 13:00:15.960297 3001 ts_tablet_manager.cc:1397] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 13:00:15.960552 3002 log.cc:824] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:15.961980 3003 tablet_bootstrap.cc:654] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:15.963285 3003 log.cc:824] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:15.964219 3002 tablet_bootstrap.cc:492] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: No bootstrap required, opened a new log
I20260619 13:00:15.964493 3002 ts_tablet_manager.cc:1397] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 13:00:15.966449 3003 tablet_bootstrap.cc:492] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: No bootstrap required, opened a new log
I20260619 13:00:15.967195 3003 ts_tablet_manager.cc:1397] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260619 13:00:15.968699 3001 raft_consensus.cc:348] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } }
I20260619 13:00:15.969064 3001 raft_consensus.cc:374] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:15.969277 3001 raft_consensus.cc:729] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb196b5a55243ab99f82ce7a8a8197e, State: Initialized, Role: FOLLOWER
I20260619 13:00:15.970005 3001 consensus_queue.cc:260] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } }
I20260619 13:00:15.975829 3002 raft_consensus.cc:348] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } }
I20260619 13:00:15.976310 3002 raft_consensus.cc:374] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:15.976426 3002 raft_consensus.cc:729] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859c3f3212a242ecb661b3e5ad5b5e3c, State: Initialized, Role: FOLLOWER
I20260619 13:00:15.977115 2843 heartbeater.cc:503] Master 127.1.87.126:42115 was elected leader, sending a full tablet report...
I20260619 13:00:15.977174 3002 consensus_queue.cc:260] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } }
I20260619 13:00:15.978569 3001 ts_tablet_manager.cc:1428] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260619 13:00:15.978583 3003 raft_consensus.cc:348] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } }
I20260619 13:00:15.979161 3003 raft_consensus.cc:374] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:15.979267 3003 raft_consensus.cc:729] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d483c83583b41bd8b70a84e5170fda4, State: Initialized, Role: FOLLOWER
I20260619 13:00:15.980161 3003 consensus_queue.cc:260] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } }
I20260619 13:00:15.980705 2982 heartbeater.cc:503] Master 127.1.87.126:42115 was elected leader, sending a full tablet report...
I20260619 13:00:15.982583 3002 ts_tablet_manager.cc:1428] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260619 13:00:15.985186 3003 ts_tablet_manager.cc:1428] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260619 13:00:16.013275 3007 raft_consensus.cc:482] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:16.013567 3007 raft_consensus.cc:504] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } }
I20260619 13:00:16.015825 3007 leader_election.cc:288] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 859c3f3212a242ecb661b3e5ad5b5e3c (127.1.87.67:40883), 7d483c83583b41bd8b70a84e5170fda4 (127.1.87.65:45399)
I20260619 13:00:16.023229 2997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.016427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:56241 (local address 127.1.87.67:40883)
0619 13:00:16.016671 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.016679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.017371 (+ 692us) server_negotiation.cc:407] Connection header received
0619 13:00:16.017593 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.017602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.017703 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.017770 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:16.018476 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.019191 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.021114 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.021869 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.022019 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.022526 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.022862 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.022977 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:00:16.023386 2999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.017431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:53391 (local address 127.1.87.65:45399)
0619 13:00:16.017610 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.017615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.018063 (+ 448us) server_negotiation.cc:407] Connection header received
0619 13:00:16.018293 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.018298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.018367 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.018458 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:16.019009 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.019904 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.021682 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.022436 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.022590 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.022695 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.022987 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.023107 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 13:00:16.023430 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.016425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:40883 (local address 127.1.87.66:56241)
0619 13:00:16.017226 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:16.017245 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:16.017386 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:16.017978 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:16.017981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:16.018000 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:16.018338 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.018345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.019343 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.019351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.020955 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.020965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.022526 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.022533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.022675 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.022698 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:00:16.022823 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 13:00:16.022951 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":3,"spinlock_wait_cycles":36608,"thread_start_us":476,"threads_started":1}
I20260619 13:00:16.023453 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.017310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:45399 (local address 127.1.87.66:53391)
0619 13:00:16.017953 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:16.017966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:16.018145 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:16.018483 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:16.018487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:16.018513 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:16.018850 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.018858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.020028 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.020036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.021537 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.021548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.022662 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.022668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.022828 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.022850 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:00:16.022955 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 13:00:16.023079 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":403,"threads_started":1}
I20260619 13:00:16.024434 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130d714f1584404086ca68006968c92e" candidate_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d483c83583b41bd8b70a84e5170fda4" is_pre_election: true
I20260619 13:00:16.024434 2937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130d714f1584404086ca68006968c92e" candidate_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" is_pre_election: true
I20260619 13:00:16.024984 2659 raft_consensus.cc:2454] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb196b5a55243ab99f82ce7a8a8197e in term 0.
I20260619 13:00:16.024984 2937 raft_consensus.cc:2454] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb196b5a55243ab99f82ce7a8a8197e in term 0.
I20260619 13:00:16.025657 2727 leader_election.cc:302] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb196b5a55243ab99f82ce7a8a8197e, 859c3f3212a242ecb661b3e5ad5b5e3c; no voters:
I20260619 13:00:16.025985 3007 raft_consensus.cc:2781] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:16.026154 3007 raft_consensus.cc:482] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:16.026270 3007 raft_consensus.cc:3037] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:16.028422 3007 raft_consensus.cc:504] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } }
I20260619 13:00:16.028985 3007 leader_election.cc:288] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [CANDIDATE]: Term 1 election: Requested vote from peers 859c3f3212a242ecb661b3e5ad5b5e3c (127.1.87.67:40883), 7d483c83583b41bd8b70a84e5170fda4 (127.1.87.65:45399)
I20260619 13:00:16.029487 2937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130d714f1584404086ca68006968c92e" candidate_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c"
I20260619 13:00:16.029526 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130d714f1584404086ca68006968c92e" candidate_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d483c83583b41bd8b70a84e5170fda4"
I20260619 13:00:16.029709 2659 raft_consensus.cc:3037] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:16.029711 2937 raft_consensus.cc:3037] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:16.032941 2937 raft_consensus.cc:2454] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb196b5a55243ab99f82ce7a8a8197e in term 1.
I20260619 13:00:16.032941 2659 raft_consensus.cc:2454] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb196b5a55243ab99f82ce7a8a8197e in term 1.
I20260619 13:00:16.033311 2727 leader_election.cc:302] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb196b5a55243ab99f82ce7a8a8197e, 859c3f3212a242ecb661b3e5ad5b5e3c; no voters:
I20260619 13:00:16.033627 3007 raft_consensus.cc:2781] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:16.034475 3007 raft_consensus.cc:686] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 LEADER]: Becoming Leader. State: Replica: 0bb196b5a55243ab99f82ce7a8a8197e, State: Running, Role: LEADER
I20260619 13:00:16.034940 3007 consensus_queue.cc:237] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } }
I20260619 13:00:16.039361 2496 catalog_manager.cc:5495] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bb196b5a55243ab99f82ce7a8a8197e (127.1.87.66). New cstate: current_term: 1 leader_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } health_report { overall_health: UNKNOWN } } }
W20260619 13:00:16.052865 2705 tablet.cc:2369] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:00:16.079813 2844 tablet.cc:2369] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:00:16.088766 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.083557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.1.87.66:36919)
0619 13:00:16.083732 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.083736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.083859 (+ 123us) server_negotiation.cc:407] Connection header received
0619 13:00:16.084086 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.084090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.084159 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.084272 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:16.084581 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.085275 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.086600 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.086971 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.087058 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.087650 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:16.088134 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:16.088295 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.088431 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.088510 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 13:00:16.088832 2983 tablet.cc:2369] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:00:16.096722 2659 raft_consensus.cc:1261] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [term 1 FOLLOWER]: Refusing update from remote peer 0bb196b5a55243ab99f82ce7a8a8197e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:16.096733 2937 raft_consensus.cc:1261] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Refusing update from remote peer 0bb196b5a55243ab99f82ce7a8a8197e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:16.097612 3012 consensus_queue.cc:1035] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [LEADER]: Connected to new peer: Peer: permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:16.097961 3007 consensus_queue.cc:1035] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:16.126367 2496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:16.126612 2496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:16.129858 2496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2: Sending DeleteTablet for 3 replicas of tablet 130d714f1584404086ca68006968c92e
I20260619 13:00:16.130990 2773 tablet_service.cc:1514] Processing DeleteTablet for tablet 130d714f1584404086ca68006968c92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:16 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:45036
I20260619 13:00:16.131306 2912 tablet_service.cc:1514] Processing DeleteTablet for tablet 130d714f1584404086ca68006968c92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:16 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:44014
I20260619 13:00:16.132061 2634 tablet_service.cc:1514] Processing DeleteTablet for tablet 130d714f1584404086ca68006968c92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:16 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:48210
I20260619 13:00:16.132372 2496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:00:16.132998 2496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:16.134456 3027 tablet_replica.cc:330] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: stopping tablet replica
I20260619 13:00:16.134877 3027 raft_consensus.cc:2229] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:16.135360 3027 raft_consensus.cc:2258] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:16.136091 3026 tablet_replica.cc:330] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: stopping tablet replica
I20260619 13:00:16.137317 3026 raft_consensus.cc:2229] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:16.137547 3027 ts_tablet_manager.cc:1905] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:16.137835 3026 raft_consensus.cc:2258] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:16.138592 3025 tablet_replica.cc:330] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: stopping tablet replica
I20260619 13:00:16.138999 3025 raft_consensus.cc:2229] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:00:16.141031 3026 ts_tablet_manager.cc:1905] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:16.141609 2634 tablet_service.cc:1467] Processing CreateTablet for tablet e9ba092bdddf43319a4f05db4951f1f5 (DEFAULT_TABLE table=test-table [id=d0a7ca06e7444c62a4a39502846ecda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:16.142047 2634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ba092bdddf43319a4f05db4951f1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:16.142206 2772 tablet_service.cc:1467] Processing CreateTablet for tablet e9ba092bdddf43319a4f05db4951f1f5 (DEFAULT_TABLE table=test-table [id=d0a7ca06e7444c62a4a39502846ecda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:16.142752 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ba092bdddf43319a4f05db4951f1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:16.146726 3003 tablet_bootstrap.cc:492] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: Bootstrap starting.
I20260619 13:00:16.148044 2912 tablet_service.cc:1467] Processing CreateTablet for tablet e9ba092bdddf43319a4f05db4951f1f5 (DEFAULT_TABLE table=test-table [id=d0a7ca06e7444c62a4a39502846ecda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:16.148439 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ba092bdddf43319a4f05db4951f1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:16.148697 3003 tablet_bootstrap.cc:654] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:16.155481 3002 tablet_bootstrap.cc:492] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: Bootstrap starting.
I20260619 13:00:16.156731 3025 raft_consensus.cc:2258] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:16.157478 3002 tablet_bootstrap.cc:654] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:16.157917 3003 tablet_bootstrap.cc:492] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: No bootstrap required, opened a new log
I20260619 13:00:16.158082 3003 ts_tablet_manager.cc:1397] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20260619 13:00:16.158489 3001 tablet_bootstrap.cc:492] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: Bootstrap starting.
I20260619 13:00:16.158736 3025 ts_tablet_manager.cc:1905] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:16.158915 3003 raft_consensus.cc:348] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.159137 3003 raft_consensus.cc:374] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:16.159251 3003 raft_consensus.cc:729] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d483c83583b41bd8b70a84e5170fda4, State: Initialized, Role: FOLLOWER
I20260619 13:00:16.159412 3027 ts_tablet_manager.cc:1918] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:00:16.159626 3027 log.cc:1196] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/130d714f1584404086ca68006968c92e
I20260619 13:00:16.159580 3003 consensus_queue.cc:260] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.160306 3001 tablet_bootstrap.cc:654] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:16.160326 3027 ts_tablet_manager.cc:1939] T 130d714f1584404086ca68006968c92e P 7d483c83583b41bd8b70a84e5170fda4: Deleting consensus metadata
I20260619 13:00:16.160882 3003 ts_tablet_manager.cc:1428] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 13:00:16.163632 2484 catalog_manager.cc:4841] TS 7d483c83583b41bd8b70a84e5170fda4 (127.1.87.65:45399): tablet 130d714f1584404086ca68006968c92e (table test-table [id=b4cfe1ea88e34b09a17175314bbf2950]) successfully deleted
I20260619 13:00:16.164717 3026 ts_tablet_manager.cc:1918] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:00:16.164979 3026 log.cc:1196] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/130d714f1584404086ca68006968c92e
I20260619 13:00:16.166074 3026 ts_tablet_manager.cc:1939] T 130d714f1584404086ca68006968c92e P 859c3f3212a242ecb661b3e5ad5b5e3c: Deleting consensus metadata
I20260619 13:00:16.168670 2484 catalog_manager.cc:4841] TS 859c3f3212a242ecb661b3e5ad5b5e3c (127.1.87.67:40883): tablet 130d714f1584404086ca68006968c92e (table test-table [id=b4cfe1ea88e34b09a17175314bbf2950]) successfully deleted
I20260619 13:00:16.169752 3025 ts_tablet_manager.cc:1918] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:00:16.169910 3025 log.cc:1196] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/130d714f1584404086ca68006968c92e
I20260619 13:00:16.170434 3025 ts_tablet_manager.cc:1939] T 130d714f1584404086ca68006968c92e P 0bb196b5a55243ab99f82ce7a8a8197e: Deleting consensus metadata
I20260619 13:00:16.170477 3002 tablet_bootstrap.cc:492] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: No bootstrap required, opened a new log
I20260619 13:00:16.170627 3002 ts_tablet_manager.cc:1397] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20260619 13:00:16.171622 3002 raft_consensus.cc:348] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.171901 3002 raft_consensus.cc:374] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:16.171999 3002 raft_consensus.cc:729] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859c3f3212a242ecb661b3e5ad5b5e3c, State: Initialized, Role: FOLLOWER
I20260619 13:00:16.172116 2482 catalog_manager.cc:4841] TS 0bb196b5a55243ab99f82ce7a8a8197e (127.1.87.66:36919): tablet 130d714f1584404086ca68006968c92e (table test-table [id=b4cfe1ea88e34b09a17175314bbf2950]) successfully deleted
I20260619 13:00:16.172276 3002 consensus_queue.cc:260] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.174576 3002 ts_tablet_manager.cc:1428] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260619 13:00:16.175418 3001 tablet_bootstrap.cc:492] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: No bootstrap required, opened a new log
I20260619 13:00:16.175575 3001 ts_tablet_manager.cc:1397] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.007s
I20260619 13:00:16.176502 3001 raft_consensus.cc:348] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.176733 3001 raft_consensus.cc:374] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:16.177253 3001 raft_consensus.cc:729] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb196b5a55243ab99f82ce7a8a8197e, State: Initialized, Role: FOLLOWER
I20260619 13:00:16.177547 3001 consensus_queue.cc:260] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.178149 3001 ts_tablet_manager.cc:1428] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 13:00:16.188257 3009 raft_consensus.cc:482] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:16.188553 3009 raft_consensus.cc:504] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.190119 3009 leader_election.cc:288] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bb196b5a55243ab99f82ce7a8a8197e (127.1.87.66:36919), 859c3f3212a242ecb661b3e5ad5b5e3c (127.1.87.67:40883)
I20260619 13:00:16.198418 2997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.190549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:50941 (local address 127.1.87.67:40883)
0619 13:00:16.190811 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.190818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.191930 (+ 1112us) server_negotiation.cc:407] Connection header received
0619 13:00:16.192161 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.192165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.192233 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.192330 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:16.192925 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.194044 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.196159 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.196901 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.197057 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.197543 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.198035 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.198159 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260619 13:00:16.198472 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.190538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:40883 (local address 127.1.87.65:50941)
0619 13:00:16.191829 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:16.191846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:16.192013 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:16.192344 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:16.192348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:16.192374 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:16.192748 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.192758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.194190 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.194194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.196026 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.196035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.197638 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.197649 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.197779 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.197806 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 13:00:16.198020 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 13:00:16.198148 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"thread_start_us":1026,"threads_started":1}
I20260619 13:00:16.198525 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.190957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:42263 (local address 127.1.87.66:36919)
0619 13:00:16.191482 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.191492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.191510 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:00:16.191573 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.191578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.191667 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.191760 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:16.192575 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.193438 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.195769 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.197036 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.197172 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.197271 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.198070 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.198197 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404}
I20260619 13:00:16.199463 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.190430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:36919 (local address 127.1.87.65:42263)
0619 13:00:16.191088 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:16.191113 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:16.191266 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:16.191939 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:16.191945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:16.191971 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:16.192331 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.192343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.193578 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.193585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.195611 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.195624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.197675 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.197683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.197881 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.197918 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:00:16.198946 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0619 13:00:16.199080 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"spinlock_wait_cycles":5376,"thread_start_us":1320,"threads_started":1}
I20260619 13:00:16.199438 2937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ba092bdddf43319a4f05db4951f1f5" candidate_uuid: "7d483c83583b41bd8b70a84e5170fda4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" is_pre_election: true
I20260619 13:00:16.199690 2937 raft_consensus.cc:2454] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d483c83583b41bd8b70a84e5170fda4 in term 0.
I20260619 13:00:16.200289 2587 leader_election.cc:302] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d483c83583b41bd8b70a84e5170fda4, 859c3f3212a242ecb661b3e5ad5b5e3c; no voters:
I20260619 13:00:16.200273 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ba092bdddf43319a4f05db4951f1f5" candidate_uuid: "7d483c83583b41bd8b70a84e5170fda4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" is_pre_election: true
I20260619 13:00:16.200728 2798 raft_consensus.cc:2454] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d483c83583b41bd8b70a84e5170fda4 in term 0.
I20260619 13:00:16.200980 3009 raft_consensus.cc:2781] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:16.201093 3009 raft_consensus.cc:482] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:16.201184 3009 raft_consensus.cc:3037] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:16.204272 3009 raft_consensus.cc:504] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.204991 3009 leader_election.cc:288] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [CANDIDATE]: Term 1 election: Requested vote from peers 0bb196b5a55243ab99f82ce7a8a8197e (127.1.87.66:36919), 859c3f3212a242ecb661b3e5ad5b5e3c (127.1.87.67:40883)
I20260619 13:00:16.205595 2937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ba092bdddf43319a4f05db4951f1f5" candidate_uuid: "7d483c83583b41bd8b70a84e5170fda4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c"
I20260619 13:00:16.205821 2937 raft_consensus.cc:3037] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:16.207744 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ba092bdddf43319a4f05db4951f1f5" candidate_uuid: "7d483c83583b41bd8b70a84e5170fda4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb196b5a55243ab99f82ce7a8a8197e"
I20260619 13:00:16.207948 2798 raft_consensus.cc:3037] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:16.208473 2937 raft_consensus.cc:2454] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d483c83583b41bd8b70a84e5170fda4 in term 1.
I20260619 13:00:16.208884 2587 leader_election.cc:302] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d483c83583b41bd8b70a84e5170fda4, 859c3f3212a242ecb661b3e5ad5b5e3c; no voters:
I20260619 13:00:16.209144 3009 raft_consensus.cc:2781] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:16.209971 3009 raft_consensus.cc:686] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 1 LEADER]: Becoming Leader. State: Replica: 7d483c83583b41bd8b70a84e5170fda4, State: Running, Role: LEADER
I20260619 13:00:16.210381 3009 consensus_queue.cc:237] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.211165 2798 raft_consensus.cc:2454] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d483c83583b41bd8b70a84e5170fda4 in term 1.
I20260619 13:00:16.213896 2497 catalog_manager.cc:5495] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d483c83583b41bd8b70a84e5170fda4 (127.1.87.65). New cstate: current_term: 1 leader_uuid: "7d483c83583b41bd8b70a84e5170fda4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } health_report { overall_health: UNKNOWN } } }
Jun 19 13:00:16 dist-test-slave-tpsm krb5kdc[2452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:16.265424 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.250252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.1.87.126:42115)
0619 13:00:16.250446 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.250458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.251161 (+ 703us) server_negotiation.cc:407] Connection header received
0619 13:00:16.251378 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.251383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.251487 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.251580 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:16.252852 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.253572 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.255211 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.255725 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.255864 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.259930 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:16.259934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:16.259938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:16.259967 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:16.262645 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:16.263456 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:16.263463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:16.263471 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:16.263590 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:16.263953 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:16.263958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:16.263961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:16.264223 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:16.264392 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.264801 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.265137 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":62}
I20260619 13:00:16.265460 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.250273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42115 (local address 127.0.0.1:49292)
0619 13:00:16.251053 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:16.251069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:16.251237 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:16.251603 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:16.251608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:16.252381 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:16.252715 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.252726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.253692 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.253700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.255072 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.255083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.255899 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.255904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.256073 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.256961 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:16.256987 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:16.259752 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:16.262882 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:16.262893 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:16.262901 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:16.263304 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:16.263700 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:16.263706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:16.263710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:16.263844 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:16.264367 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:16.264375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:16.264549 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 13:00:16.264707 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 13:00:16.265006 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":419,"threads_started":1}
I20260619 13:00:16.270787 2497 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:49292:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 13:00:16.271862 2497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:16.279038 2634 tablet_service.cc:1467] Processing CreateTablet for tablet 5693ed11f45745a292ee1d27c309e885 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6e4ef23a19c449e8578311b4fc041b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:00:16.279254 2772 tablet_service.cc:1467] Processing CreateTablet for tablet 5693ed11f45745a292ee1d27c309e885 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6e4ef23a19c449e8578311b4fc041b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:00:16.279671 2634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5693ed11f45745a292ee1d27c309e885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:16.279364 2912 tablet_service.cc:1467] Processing CreateTablet for tablet 5693ed11f45745a292ee1d27c309e885 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6e4ef23a19c449e8578311b4fc041b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:00:16.280491 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5693ed11f45745a292ee1d27c309e885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:16.280717 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5693ed11f45745a292ee1d27c309e885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:16.287405 3003 tablet_bootstrap.cc:492] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: Bootstrap starting.
I20260619 13:00:16.288647 3002 tablet_bootstrap.cc:492] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: Bootstrap starting.
I20260619 13:00:16.289052 3003 tablet_bootstrap.cc:654] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:16.290504 3002 tablet_bootstrap.cc:654] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:16.293829 3001 tablet_bootstrap.cc:492] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: Bootstrap starting.
I20260619 13:00:16.296893 3001 tablet_bootstrap.cc:654] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:16.300719 3003 tablet_bootstrap.cc:492] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: No bootstrap required, opened a new log
I20260619 13:00:16.300918 3003 ts_tablet_manager.cc:1397] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260619 13:00:16.301731 3003 raft_consensus.cc:348] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.301992 3003 raft_consensus.cc:374] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:16.302093 3003 raft_consensus.cc:729] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d483c83583b41bd8b70a84e5170fda4, State: Initialized, Role: FOLLOWER
I20260619 13:00:16.302520 3002 tablet_bootstrap.cc:492] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: No bootstrap required, opened a new log
I20260619 13:00:16.302371 3003 consensus_queue.cc:260] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.302759 3002 ts_tablet_manager.cc:1397] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260619 13:00:16.302784 3009 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.303064 3009 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:16.303699 3002 raft_consensus.cc:348] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.304000 3002 raft_consensus.cc:374] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:16.304210 3002 raft_consensus.cc:729] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859c3f3212a242ecb661b3e5ad5b5e3c, State: Initialized, Role: FOLLOWER
I20260619 13:00:16.304548 3002 consensus_queue.cc:260] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.305269 3002 ts_tablet_manager.cc:1428] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:00:16.305078 3008 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.305838 3003 ts_tablet_manager.cc:1428] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260619 13:00:16.305985 3008 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:16.313055 3001 tablet_bootstrap.cc:492] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: No bootstrap required, opened a new log
I20260619 13:00:16.313256 3001 ts_tablet_manager.cc:1397] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.010s
I20260619 13:00:16.314080 3001 raft_consensus.cc:348] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.314357 3001 raft_consensus.cc:374] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:16.314446 3001 raft_consensus.cc:729] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb196b5a55243ab99f82ce7a8a8197e, State: Initialized, Role: FOLLOWER
I20260619 13:00:16.314738 3001 consensus_queue.cc:260] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.315183 3007 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.315493 3007 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:16.316180 3001 ts_tablet_manager.cc:1428] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 13:00:16.616999 3009 consensus_queue.cc:1035] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:16.624006 3047 consensus_queue.cc:1035] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:16.742055 3008 raft_consensus.cc:482] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:16.742358 3047 raft_consensus.cc:482] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:16.742549 3047 raft_consensus.cc:504] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.742496 3008 raft_consensus.cc:504] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.743220 3047 leader_election.cc:288] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bb196b5a55243ab99f82ce7a8a8197e (127.1.87.66:36919), 859c3f3212a242ecb661b3e5ad5b5e3c (127.1.87.67:40883)
I20260619 13:00:16.743791 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5693ed11f45745a292ee1d27c309e885" candidate_uuid: "7d483c83583b41bd8b70a84e5170fda4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" is_pre_election: true
I20260619 13:00:16.743854 2937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5693ed11f45745a292ee1d27c309e885" candidate_uuid: "7d483c83583b41bd8b70a84e5170fda4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" is_pre_election: true
I20260619 13:00:16.744061 2937 raft_consensus.cc:2454] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d483c83583b41bd8b70a84e5170fda4 in term 0.
I20260619 13:00:16.744277 3008 leader_election.cc:288] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d483c83583b41bd8b70a84e5170fda4 (127.1.87.65:45399), 0bb196b5a55243ab99f82ce7a8a8197e (127.1.87.66:36919)
I20260619 13:00:16.744508 2587 leader_election.cc:302] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d483c83583b41bd8b70a84e5170fda4, 859c3f3212a242ecb661b3e5ad5b5e3c; no voters:
I20260619 13:00:16.744037 2798 raft_consensus.cc:2454] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d483c83583b41bd8b70a84e5170fda4 in term 0.
I20260619 13:00:16.744803 3047 raft_consensus.cc:2781] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:16.744915 3047 raft_consensus.cc:482] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:16.745080 3047 raft_consensus.cc:3037] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:16.749963 3047 raft_consensus.cc:504] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.751374 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5693ed11f45745a292ee1d27c309e885" candidate_uuid: "7d483c83583b41bd8b70a84e5170fda4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb196b5a55243ab99f82ce7a8a8197e"
I20260619 13:00:16.751729 2798 raft_consensus.cc:3037] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:16.756402 3047 leader_election.cc:288] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [CANDIDATE]: Term 1 election: Requested vote from peers 0bb196b5a55243ab99f82ce7a8a8197e (127.1.87.66:36919), 859c3f3212a242ecb661b3e5ad5b5e3c (127.1.87.67:40883)
I20260619 13:00:16.756879 2937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5693ed11f45745a292ee1d27c309e885" candidate_uuid: "7d483c83583b41bd8b70a84e5170fda4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c"
I20260619 13:00:16.757046 2798 raft_consensus.cc:2454] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d483c83583b41bd8b70a84e5170fda4 in term 1.
I20260619 13:00:16.757157 2937 raft_consensus.cc:3037] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:16.757741 2585 leader_election.cc:302] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb196b5a55243ab99f82ce7a8a8197e, 7d483c83583b41bd8b70a84e5170fda4; no voters:
I20260619 13:00:16.758031 3047 raft_consensus.cc:2781] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:16.758191 3047 raft_consensus.cc:686] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [term 1 LEADER]: Becoming Leader. State: Replica: 7d483c83583b41bd8b70a84e5170fda4, State: Running, Role: LEADER
I20260619 13:00:16.758453 3047 consensus_queue.cc:237] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } }
I20260619 13:00:16.760285 2937 raft_consensus.cc:2454] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d483c83583b41bd8b70a84e5170fda4 in term 1.
I20260619 13:00:16.760401 3009 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: TxnStatusTablet state changed. Reason: New leader 7d483c83583b41bd8b70a84e5170fda4. Latest consensus state: current_term: 1 leader_uuid: "7d483c83583b41bd8b70a84e5170fda4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.760676 3009 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:00:16.761278 3055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 13:00:16.761498 3055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 13:00:16.762566 2496 catalog_manager.cc:5495] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d483c83583b41bd8b70a84e5170fda4 (127.1.87.65). New cstate: current_term: 1 leader_uuid: "7d483c83583b41bd8b70a84e5170fda4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } health_report { overall_health: UNKNOWN } } }
I20260619 13:00:16.764595 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.744623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:36919 (local address 127.1.87.67:39083)
0619 13:00:16.755927 (+ 11304us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:16.755945 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:16.756067 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:16.757334 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:16.757339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:16.757371 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:16.757684 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.757695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.759027 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.759032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.761122 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.761136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.763345 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.763352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.763535 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.763574 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 13:00:16.763665 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 13:00:16.764191 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11181,"mutex_wait_us":2,"thread_start_us":10644,"threads_started":1}
I20260619 13:00:16.765336 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.747494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:39083 (local address 127.1.87.66:36919)
0619 13:00:16.756933 (+ 9439us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.756939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.756957 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:00:16.757017 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.757021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.757108 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.757214 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:16.757882 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.758883 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.761472 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.762815 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.763125 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.763234 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.764872 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.765024 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9316,"thread_start_us":9813,"threads_started":1}
I20260619 13:00:16.765604 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.744746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:52271 (local address 127.1.87.65:45399)
0619 13:00:16.755995 (+ 11249us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.756003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.756028 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:00:16.756124 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.756131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.756205 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.756292 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:16.757309 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.758688 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.762664 (+ 3976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.764400 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.764497 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.764619 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.765128 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.765268 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11122,"thread_start_us":11051,"threads_started":1}
I20260619 13:00:16.766597 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.744391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:45399 (local address 127.1.87.67:52271)
0619 13:00:16.755152 (+ 10761us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:16.755176 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:16.755953 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:16.756465 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:16.756471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:16.756592 (+ 121us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:16.756944 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.756956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.758917 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.758925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.762441 (+ 3516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.762457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.764780 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.764785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.764952 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.764984 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:00:16.765765 (+ 781us) client_negotiation.cc:240] Negotiation successful
0619 13:00:16.766209 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10610,"mutex_wait_us":1241,"thread_start_us":10490,"threads_started":1}
I20260619 13:00:16.767016 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5693ed11f45745a292ee1d27c309e885" candidate_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" is_pre_election: true
I20260619 13:00:16.767421 2798 raft_consensus.cc:2379] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 859c3f3212a242ecb661b3e5ad5b5e3c in current term 1: Already voted for candidate 7d483c83583b41bd8b70a84e5170fda4 in this term.
I20260619 13:00:16.767652 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5693ed11f45745a292ee1d27c309e885" candidate_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d483c83583b41bd8b70a84e5170fda4" is_pre_election: true
I20260619 13:00:16.768296 2866 leader_election.cc:302] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 859c3f3212a242ecb661b3e5ad5b5e3c; no voters: 0bb196b5a55243ab99f82ce7a8a8197e, 7d483c83583b41bd8b70a84e5170fda4
I20260619 13:00:16.768699 3008 raft_consensus.cc:2726] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 13:00:16.868734 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.861783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:45399 (local address 127.0.0.1:48212)
0619 13:00:16.862354 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:16.862369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:16.862542 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:16.863002 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:16.863006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:16.863017 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:00:16.863227 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.863235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.864614 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.864623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.865824 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.865838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.866975 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.866985 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.867113 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.867180 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:00:16.868263 (+ 1083us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:00:16.868286 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:00:16.868347 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 13:00:16.868458 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":337,"threads_started":1}
I20260619 13:00:16.868773 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.861854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.1.87.65:45399)
0619 13:00:16.862029 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.862034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.862458 (+ 424us) server_negotiation.cc:407] Connection header received
0619 13:00:16.862766 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.862770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.862849 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.862987 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:16.863397 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.864475 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.865974 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.866822 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.866961 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.867401 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:16.868145 (+ 744us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:16.868295 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.868372 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.868464 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:00:16.872712 2937 raft_consensus.cc:1261] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Refusing update from remote peer 7d483c83583b41bd8b70a84e5170fda4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:16.872963 2798 raft_consensus.cc:1261] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 FOLLOWER]: Refusing update from remote peer 7d483c83583b41bd8b70a84e5170fda4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:16.873380 3047 consensus_queue.cc:1035] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:16.873677 3009 consensus_queue.cc:1035] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:16.877012 3007 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: TxnStatusTablet state changed. Reason: New leader 7d483c83583b41bd8b70a84e5170fda4. Latest consensus state: current_term: 1 leader_uuid: "7d483c83583b41bd8b70a84e5170fda4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.877291 3007 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:16.878396 3008 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: TxnStatusTablet state changed. Reason: New leader 7d483c83583b41bd8b70a84e5170fda4. Latest consensus state: current_term: 1 leader_uuid: "7d483c83583b41bd8b70a84e5170fda4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.878667 3008 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:16.878861 3045 mvcc.cc:205] Tried to move back new op lower bound from 7298555973104435200 to 7298555972645396480. Current Snapshot: MvccSnapshot[applied={T|T < 7298555973104435200}]
I20260619 13:00:16.883174 3049 mvcc.cc:205] Tried to move back new op lower bound from 7298555973104435200 to 7298555972645396480. Current Snapshot: MvccSnapshot[applied={T|T < 7298555973104435200}]
I20260619 13:00:16.889272 3009 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d483c83583b41bd8b70a84e5170fda4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.889595 3009 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:00:16.889780 3008 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d483c83583b41bd8b70a84e5170fda4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.890040 3008 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 859c3f3212a242ecb661b3e5ad5b5e3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:16.892050 3009 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d483c83583b41bd8b70a84e5170fda4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.892323 3009 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 7d483c83583b41bd8b70a84e5170fda4: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:00:16.899645 3007 tablet_replica.cc:439] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d483c83583b41bd8b70a84e5170fda4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d483c83583b41bd8b70a84e5170fda4" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45399 } } peers { permanent_uuid: "0bb196b5a55243ab99f82ce7a8a8197e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 36919 } } peers { permanent_uuid: "859c3f3212a242ecb661b3e5ad5b5e3c" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 40883 } } }
I20260619 13:00:16.899991 3007 tablet_replica.cc:442] T 5693ed11f45745a292ee1d27c309e885 P 0bb196b5a55243ab99f82ce7a8a8197e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:16.911130 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.898244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.1.87.65:45399)
0619 13:00:16.898424 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.898433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.905108 (+ 6675us) server_negotiation.cc:407] Connection header received
0619 13:00:16.905245 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.905250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.905345 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.905454 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:16.906098 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.907283 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.908918 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.909594 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.909728 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.910120 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:16.910263 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:16.910354 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.910689 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.910779 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 13:00:16.928421 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.922759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:58933 (local address 127.1.87.65:45399)
0619 13:00:16.922919 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:16.922924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:16.923574 (+ 650us) server_negotiation.cc:407] Connection header received
0619 13:00:16.923797 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:16.923806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:16.923871 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:16.923978 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:16.924461 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.925226 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.926671 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.927120 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.927255 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.927706 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:16.927822 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:16.927919 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:16.928047 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 13:00:16.928112 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:00:16.928539 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:16.922797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:45399 (local address 127.1.87.65:58933)
0619 13:00:16.923476 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:16.923492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:16.923667 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:16.923988 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:16.923996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:16.924005 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:00:16.924309 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.924317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.925361 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.925369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.926520 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:16.926531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:16.927300 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:16.927309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:16.927465 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:16.927519 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:00:16.927917 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:00:16.927927 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:00:16.928020 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 13:00:16.928157 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":330,"threads_started":1}
I20260619 13:00:16.996101 2497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:16.996308 2497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:16.999571 2497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e7b3c0182484fbea33933ef878372d2: Sending DeleteTablet for 3 replicas of tablet e9ba092bdddf43319a4f05db4951f1f5
I20260619 13:00:17.000559 2634 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ba092bdddf43319a4f05db4951f1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:16 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:48210
I20260619 13:00:17.000610 2912 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ba092bdddf43319a4f05db4951f1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:16 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:44014
I20260619 13:00:17.001372 2772 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ba092bdddf43319a4f05db4951f1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:16 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:45036
I20260619 13:00:17.010681 3071 tablet_replica.cc:330] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: stopping tablet replica
I20260619 13:00:17.010970 3070 tablet_replica.cc:330] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: stopping tablet replica
I20260619 13:00:17.010927 3071 raft_consensus.cc:2229] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:17.011154 3070 raft_consensus.cc:2229] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:00:17.011231 3071 raft_consensus.cc:2258] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:17.011925 3070 raft_consensus.cc:2258] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:17.013762 3071 ts_tablet_manager.cc:1905] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:17.013765 3070 ts_tablet_manager.cc:1905] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:17.016251 3072 tablet_replica.cc:330] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: stopping tablet replica
I20260619 13:00:17.016569 3072 raft_consensus.cc:2229] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:17.016856 3072 raft_consensus.cc:2258] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:17.017901 3072 ts_tablet_manager.cc:1905] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:17.022397 3071 ts_tablet_manager.cc:1918] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:00:17.022534 3071 log.cc:1196] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/e9ba092bdddf43319a4f05db4951f1f5
I20260619 13:00:17.022941 3071 ts_tablet_manager.cc:1939] T e9ba092bdddf43319a4f05db4951f1f5 P 859c3f3212a242ecb661b3e5ad5b5e3c: Deleting consensus metadata
I20260619 13:00:17.023542 3070 ts_tablet_manager.cc:1918] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:00:17.023705 3070 log.cc:1196] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/e9ba092bdddf43319a4f05db4951f1f5
I20260619 13:00:17.024158 3070 ts_tablet_manager.cc:1939] T e9ba092bdddf43319a4f05db4951f1f5 P 7d483c83583b41bd8b70a84e5170fda4: Deleting consensus metadata
I20260619 13:00:17.024223 2484 catalog_manager.cc:4841] TS 859c3f3212a242ecb661b3e5ad5b5e3c (127.1.87.67:40883): tablet e9ba092bdddf43319a4f05db4951f1f5 (table test-table [id=d0a7ca06e7444c62a4a39502846ecda2]) successfully deleted
I20260619 13:00:17.025234 2484 catalog_manager.cc:4841] TS 7d483c83583b41bd8b70a84e5170fda4 (127.1.87.65:45399): tablet e9ba092bdddf43319a4f05db4951f1f5 (table test-table [id=d0a7ca06e7444c62a4a39502846ecda2]) successfully deleted
I20260619 13:00:17.025943 3072 ts_tablet_manager.cc:1918] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:00:17.026082 3072 log.cc:1196] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/e9ba092bdddf43319a4f05db4951f1f5
I20260619 13:00:17.026556 3072 ts_tablet_manager.cc:1939] T e9ba092bdddf43319a4f05db4951f1f5 P 0bb196b5a55243ab99f82ce7a8a8197e: Deleting consensus metadata
I20260619 13:00:17.027523 2482 catalog_manager.cc:4841] TS 0bb196b5a55243ab99f82ce7a8a8197e (127.1.87.66:36919): tablet e9ba092bdddf43319a4f05db4951f1f5 (table test-table [id=d0a7ca06e7444c62a4a39502846ecda2]) successfully deleted
Jun 19 13:00:17 dist-test-slave-tpsm krb5kdc[2452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:17.044348 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:17.015869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.1.87.65:45399)
0619 13:00:17.016052 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:17.016061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:17.025006 (+ 8945us) server_negotiation.cc:407] Connection header received
0619 13:00:17.025331 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:17.025339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:17.025781 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:17.025868 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:17.027646 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.028723 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.030102 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.030789 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.030926 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:17.036216 (+ 5290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:17.036232 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:17.036253 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:17.036314 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:17.040830 (+ 4516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.041735 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.041744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.041748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.041909 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.042423 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.042430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.042434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.042991 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:17.043358 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:17.043635 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 13:00:17.043895 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 13:00:17.045113 2699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48244
I20260619 13:00:17.065079 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:17.048110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.1.87.126:42115)
0619 13:00:17.048753 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:17.048757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:17.048773 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:00:17.048822 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:17.048825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:17.048881 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:17.048972 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:17.050998 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.052164 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.054620 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.055278 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.055449 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:17.058857 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:17.058862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:17.058865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:17.058993 (+ 128us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:17.062190 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.063120 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.063124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.063127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.063267 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.063747 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.063750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.063752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.064040 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:17.064222 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:17.064602 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 13:00:17.064792 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":322,"threads_started":1}
W20260619 13:00:17.065702 2497 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49306
I20260619 13:00:17.068480 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2568
I20260619 13:00:17.086023 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:17.070919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.1.87.65:45399)
0619 13:00:17.071261 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:17.071268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:17.071294 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:00:17.071484 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:17.071490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:17.071610 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:17.071741 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:17.072879 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.073985 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.075269 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.076086 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.076253 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:17.078536 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:17.078544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:17.078549 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:17.078591 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:17.082457 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.083208 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.083220 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.083223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.083417 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.083800 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.083805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.083809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.084158 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:17.084415 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:17.085378 (+ 963us) server_negotiation.cc:299] Negotiation successful
0619 13:00:17.085641 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
W20260619 13:00:17.086750 2699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48260
W20260619 13:00:17.087419 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2568: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:00:17.087635 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 2568
I20260619 13:00:17.105594 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2708
Jun 19 13:00:17 dist-test-slave-tpsm krb5kdc[2452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:00:17.122408 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:17.106316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.1.87.66:36919)
0619 13:00:17.106651 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:17.106657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:17.106677 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:00:17.106890 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:17.106895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:17.106980 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:17.107089 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:17.108363 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.109064 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.110289 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.110845 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.111001 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:17.115406 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:17.115423 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:17.115440 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:17.115478 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:17.119272 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.120423 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.120430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.120432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.120592 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.121016 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.121020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.121022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.121430 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:17.121672 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:17.121982 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 13:00:17.122163 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 13:00:17.123137 2838 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45054
W20260619 13:00:17.123836 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2708: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:00:17.123986 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 2708
I20260619 13:00:17.146596 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2847
Jun 19 13:00:17 dist-test-slave-tpsm krb5kdc[2452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:00:17.162881 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:17.147374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.1.87.67:40883)
0619 13:00:17.147857 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:17.147861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:17.147878 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:00:17.147935 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:17.147938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:17.147997 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:17.148095 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:17.149460 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.150496 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.151945 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:17.152426 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:17.152523 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:17.156957 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:17.156972 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:17.156988 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:17.157029 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:17.160219 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.160893 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.160900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.160902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.161055 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:17.161454 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:17.161457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:17.161460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:17.161830 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:17.162069 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:17.162401 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 13:00:17.162586 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":237,"threads_started":1}
W20260619 13:00:17.163601 2977 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44016
W20260619 13:00:17.164240 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2847: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:00:17.164377 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 2847
I20260619 13:00:17.181115 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 2465
I20260619 13:00:17.182302 2537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:17.295382 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 2465
2026-06-19T13:00:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4804 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:17 dist-test-slave-tpsm krb5kdc[3083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:17 dist-test-slave-tpsm krb5kdc[3083](info): set up 2 sockets
Jun 19 13:00:17 dist-test-slave-tpsm krb5kdc[3083](info): commencing operation
krb5kdc: starting...
W20260619 13:00:18.391443 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:18 dist-test-slave-tpsm krb5kdc[3083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:18.589706 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37575
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:45463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:37575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:18.922432 3096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:18.922883 3096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:18.923048 3096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:18.931018 3096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:18.931277 3096 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 13:00:18.931409 3096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:18.931486 3096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:18.931538 3096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:18.931595 3096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:18.941761 3096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:45463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:37575
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:18.943038 3096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:18.944288 3096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:18.956653 3101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:18.956609 3104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:18.956625 3102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:18.957327 3096 server_base.cc:1034] running on GCE node
I20260619 13:00:18.958173 3096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:18.959995 3096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:18.961232 3096 hybrid_clock.cc:648] HybridClock initialized: now 1781874018961171 us; error 38 us; skew 500 ppm
I20260619 13:00:18.961582 3096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:18 dist-test-slave-tpsm krb5kdc[3083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874018, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:18.965003 3096 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:18.966627 3096 webserver.cc:460] Webserver started at http://127.1.87.126:35909/ using document root <none> and password file <none>
I20260619 13:00:18.967870 3096 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:18.968163 3096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:18.968564 3096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:18.971637 3096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "f1493604fe2a4aa4b953f35d8127cd11"
format_stamp: "Formatted at 2026-06-19 13:00:18 on dist-test-slave-tpsm"
server_key: "6203aeb5fb5b703fd4347b685dd22e84"
server_key_iv: "4a226e7549c2d3ece189e993dfbe6e83"
server_key_version: "encryptionkey@0"
I20260619 13:00:18.972386 3096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "f1493604fe2a4aa4b953f35d8127cd11"
format_stamp: "Formatted at 2026-06-19 13:00:18 on dist-test-slave-tpsm"
server_key: "6203aeb5fb5b703fd4347b685dd22e84"
server_key_iv: "4a226e7549c2d3ece189e993dfbe6e83"
server_key_version: "encryptionkey@0"
I20260619 13:00:18.977732 3096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 13:00:18.981566 3111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:18.982939 3096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:18.983137 3096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "f1493604fe2a4aa4b953f35d8127cd11"
format_stamp: "Formatted at 2026-06-19 13:00:18 on dist-test-slave-tpsm"
server_key: "6203aeb5fb5b703fd4347b685dd22e84"
server_key_iv: "4a226e7549c2d3ece189e993dfbe6e83"
server_key_version: "encryptionkey@0"
I20260619 13:00:18.983425 3096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:18.995651 3096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:19.000083 3096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:19.000520 3096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:19.022590 3096 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:37575
I20260619 13:00:19.022614 3172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:37575 every 8 connection(s)
I20260619 13:00:19.024037 3096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:19.028926 3173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:19.031410 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 3096
I20260619 13:00:19.031649 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:19.032435 1373 external_mini_cluster.cc:1428] Setting key 4829849fd1715a15fe1e514277f804ae
I20260619 13:00:19.041582 3173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11: Bootstrap starting.
I20260619 13:00:19.045851 3173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:19.047348 3173 log.cc:824] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:00:19 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:19.052291 3173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11: No bootstrap required, opened a new log
I20260619 13:00:19.057371 3176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:19.034771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.1.87.126:37575)
0619 13:00:19.035959 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:19.035979 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:19.036041 (+ 62us) server_negotiation.cc:407] Connection header received
0619 13:00:19.038206 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:19.038236 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:19.038776 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:19.039215 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:19.040673 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.042009 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.043251 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.044151 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.044369 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:19.049236 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:19.049264 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:19.049293 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:19.049504 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:19.053256 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:19.053990 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:19.053997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:19.054006 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:19.054176 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:19.054561 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:19.054574 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:19.054576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:19.055118 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:19.055436 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:19.055901 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 13:00:19.056328 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":575,"threads_started":1}
I20260619 13:00:19.060706 3173 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1493604fe2a4aa4b953f35d8127cd11" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37575 } }
I20260619 13:00:19.061045 3173 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:19.061168 3173 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1493604fe2a4aa4b953f35d8127cd11, State: Initialized, Role: FOLLOWER
I20260619 13:00:19.061712 3173 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1493604fe2a4aa4b953f35d8127cd11" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37575 } }
I20260619 13:00:19.061910 3173 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:19.062048 3173 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:19.062206 3173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:19.065285 3173 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1493604fe2a4aa4b953f35d8127cd11" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37575 } }
I20260619 13:00:19.065835 3173 leader_election.cc:302] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1493604fe2a4aa4b953f35d8127cd11; no voters:
I20260619 13:00:19.066417 3173 leader_election.cc:288] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:19.066574 3178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:19.067260 3178 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [term 1 LEADER]: Becoming Leader. State: Replica: f1493604fe2a4aa4b953f35d8127cd11, State: Running, Role: LEADER
I20260619 13:00:19.068181 3178 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1493604fe2a4aa4b953f35d8127cd11" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37575 } }
I20260619 13:00:19.068771 3173 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:19.070994 3180 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1493604fe2a4aa4b953f35d8127cd11. Latest consensus state: current_term: 1 leader_uuid: "f1493604fe2a4aa4b953f35d8127cd11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1493604fe2a4aa4b953f35d8127cd11" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37575 } } }
I20260619 13:00:19.071384 3180 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:19.071933 3179 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1493604fe2a4aa4b953f35d8127cd11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1493604fe2a4aa4b953f35d8127cd11" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37575 } } }
I20260619 13:00:19.072196 3179 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:19.072405 3184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:19.079772 3184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:19.090821 3184 catalog_manager.cc:1347] Generated new cluster ID: ef61ef6d827a453eb61c18f5a942f5f9
I20260619 13:00:19.090950 3184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:19.120854 3184 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:19.122795 3184 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:19.131425 3184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11: Generated new TSK 0
I20260619 13:00:19.132465 3184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:00:19.182241 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37575
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:45463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:19.514336 3200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:19.514624 3200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:19.514726 3200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:19.521941 3200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:19.522063 3200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:19.522156 3200 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 13:00:19.522209 3200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:00:19.522279 3200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:19.533432 3200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:45463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.87.126:37575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:19.535068 3200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:19.536413 3200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:19.549263 3206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:19.549520 3205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:19.549314 3208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:19.550674 3200 server_base.cc:1034] running on GCE node
I20260619 13:00:19.551255 3200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:19.552304 3200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:19.553509 3200 hybrid_clock.cc:648] HybridClock initialized: now 1781874019553445 us; error 49 us; skew 500 ppm
I20260619 13:00:19.553836 3200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:19 dist-test-slave-tpsm krb5kdc[3083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:19.557045 3200 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:00:19.558580 3200 webserver.cc:460] Webserver started at http://127.1.87.65:39717/ using document root <none> and password file <none>
I20260619 13:00:19.559288 3200 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:19.559505 3200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:19.559834 3200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:19.562681 3200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "eeb54da8bae14050a9d1f5c2d60d9e97"
format_stamp: "Formatted at 2026-06-19 13:00:19 on dist-test-slave-tpsm"
server_key: "5de66390f0de742429eab8a03bedef4b"
server_key_iv: "949da220dab38d94d788e369375f831e"
server_key_version: "encryptionkey@0"
I20260619 13:00:19.563555 3200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "eeb54da8bae14050a9d1f5c2d60d9e97"
format_stamp: "Formatted at 2026-06-19 13:00:19 on dist-test-slave-tpsm"
server_key: "5de66390f0de742429eab8a03bedef4b"
server_key_iv: "949da220dab38d94d788e369375f831e"
server_key_version: "encryptionkey@0"
I20260619 13:00:19.568800 3200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:00:19.572501 3215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:19.573879 3200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:00:19.574046 3200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "eeb54da8bae14050a9d1f5c2d60d9e97"
format_stamp: "Formatted at 2026-06-19 13:00:19 on dist-test-slave-tpsm"
server_key: "5de66390f0de742429eab8a03bedef4b"
server_key_iv: "949da220dab38d94d788e369375f831e"
server_key_version: "encryptionkey@0"
I20260619 13:00:19.574283 3200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:19.594518 3200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:19.599159 3200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:19.599498 3200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:19.602185 3200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:19.602330 3200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:19.602454 3200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:19.602550 3200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:00:19 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:19.629647 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:19.603175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:41749 (local address 127.1.87.126:37575)
0619 13:00:19.603628 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:19.603634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:19.605028 (+ 1394us) server_negotiation.cc:407] Connection header received
0619 13:00:19.608386 (+ 3358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:19.608394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:19.608506 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:19.608646 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:19.611542 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.612570 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.615819 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.616448 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.616598 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:19.621547 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:19.621552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:19.621555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:19.621591 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:19.625328 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:19.626344 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:19.626350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:19.626353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:19.626516 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:19.627558 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:19.627562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:19.627564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:19.627852 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:19.627973 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:19.629202 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0619 13:00:19.629389 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":237,"threads_started":1}
I20260619 13:00:19.631974 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:19.603892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37575 (local address 127.1.87.65:41749)
0619 13:00:19.604831 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:19.604892 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:19.607980 (+ 3088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:19.608969 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:19.608984 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:19.610064 (+ 1080us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:19.611247 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:19.611286 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.612773 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.612781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:19.615611 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:19.615631 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.616739 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.616750 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:19.616971 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:19.617883 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:19.617935 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:19.621264 (+ 3329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:19.625518 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:19.625533 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:19.625560 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:19.626165 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:19.626970 (+ 805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:19.626982 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:19.626988 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:19.627396 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:19.627999 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:19.628011 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:19.628697 (+ 686us) client_negotiation.cc:769] Sending connection context
0619 13:00:19.629092 (+ 395us) client_negotiation.cc:240] Negotiation successful
0619 13:00:19.629870 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":410,"threads_started":1}
I20260619 13:00:19.650044 3200 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:40749
I20260619 13:00:19.650080 3335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:40749 every 8 connection(s)
I20260619 13:00:19.651401 3200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:19.653579 3336 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37575
I20260619 13:00:19.653820 3336 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:19.654533 3336 heartbeater.cc:511] Master 127.1.87.126:37575 requested a full tablet report, sending...
I20260619 13:00:19.655156 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:19.641596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:39687 (local address 127.1.87.126:37575)
0619 13:00:19.641835 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:19.641844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:19.642513 (+ 669us) server_negotiation.cc:407] Connection header received
0619 13:00:19.642712 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:19.642722 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:19.642822 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:19.642935 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:19.644366 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.645050 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.646346 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.646748 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.646893 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:19.649158 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:19.649167 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:19.649175 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:19.649220 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:19.652206 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:19.653108 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:19.653113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:19.653117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:19.653299 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:19.653732 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:19.653739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:19.653742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:19.654010 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:19.654178 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:19.654581 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 13:00:19.654797 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 13:00:19.655295 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:19.641612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37575 (local address 127.1.87.65:39687)
0619 13:00:19.642366 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:19.642400 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:19.642575 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:19.642983 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:19.642990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:19.643735 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:19.644211 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:19.644222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.645199 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.645205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:19.646204 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:19.646215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:19.646940 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:19.646947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:19.647134 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:19.647908 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:19.647927 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:19.648708 (+ 781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:19.652375 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:19.652380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:19.652387 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:19.652978 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:19.653459 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:19.653463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:19.653468 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:19.653623 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:19.654144 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:19.654150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:19.654333 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 13:00:19.654481 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 13:00:19.654818 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":402,"threads_started":1}
I20260619 13:00:19.656821 3127 ts_manager.cc:194] Registered new tserver with Master: eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749)
I20260619 13:00:19.658442 3127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:41749
I20260619 13:00:19.661464 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 3200
I20260619 13:00:19.661613 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:19.662173 1373 external_mini_cluster.cc:1428] Setting key 77cc49badaf45e0e03c0928a11c7c561
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:00:19.700016 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37575
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:45463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:20.032805 3340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:20.033164 3340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:20.033313 3340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:20.044402 3340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:20.044565 3340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:20.044718 3340 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 13:00:20.044804 3340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:00:20.044936 3340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:20.054895 3340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:45463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.87.126:37575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:20.056270 3340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:20.057530 3340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:20.070062 3345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:20.070111 3346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:20.070103 3348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:20.071434 3340 server_base.cc:1034] running on GCE node
I20260619 13:00:20.072068 3340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:20.073143 3340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:20.074393 3340 hybrid_clock.cc:648] HybridClock initialized: now 1781874020074364 us; error 68 us; skew 500 ppm
I20260619 13:00:20.074723 3340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:20 dist-test-slave-tpsm krb5kdc[3083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:20.078181 3340 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:00:20.079947 3340 webserver.cc:460] Webserver started at http://127.1.87.66:44119/ using document root <none> and password file <none>
I20260619 13:00:20.080595 3340 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:20.080705 3340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:20.080962 3340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:20.083914 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "6796dd76bd9740ae844e6f681546cd0c"
format_stamp: "Formatted at 2026-06-19 13:00:20 on dist-test-slave-tpsm"
server_key: "76f2cf42859c5f8767f45ad2586133d2"
server_key_iv: "d9528a61fb8948f6d0c101e9182a6f3e"
server_key_version: "encryptionkey@0"
I20260619 13:00:20.084753 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "6796dd76bd9740ae844e6f681546cd0c"
format_stamp: "Formatted at 2026-06-19 13:00:20 on dist-test-slave-tpsm"
server_key: "76f2cf42859c5f8767f45ad2586133d2"
server_key_iv: "d9528a61fb8948f6d0c101e9182a6f3e"
server_key_version: "encryptionkey@0"
I20260619 13:00:20.090004 3340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:00:20.093652 3355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:20.095052 3340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:20.095249 3340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "6796dd76bd9740ae844e6f681546cd0c"
format_stamp: "Formatted at 2026-06-19 13:00:20 on dist-test-slave-tpsm"
server_key: "76f2cf42859c5f8767f45ad2586133d2"
server_key_iv: "d9528a61fb8948f6d0c101e9182a6f3e"
server_key_version: "encryptionkey@0"
I20260619 13:00:20.095538 3340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:20.115451 3340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:20.119957 3340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:20.120287 3340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:20.123129 3340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:20.123237 3340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:20.123397 3340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:20.123507 3340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 13:00:20 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:20.149897 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.124315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:56573 (local address 127.1.87.126:37575)
0619 13:00:20.124484 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:20.124489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:20.126518 (+ 2029us) server_negotiation.cc:407] Connection header received
0619 13:00:20.129724 (+ 3206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:20.129735 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:20.129883 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:20.130008 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:20.132324 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.133632 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.136328 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.136913 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.137074 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.142470 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:20.142480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:20.142486 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:20.142537 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:20.145720 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.146903 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.146907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.146909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.147032 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.147708 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.147714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.147717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.148057 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:20.148220 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:20.149351 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0619 13:00:20.149578 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:00:20.151440 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.125095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37575 (local address 127.1.87.66:56573)
0619 13:00:20.126248 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:20.126293 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:20.129274 (+ 2981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:20.130330 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:20.130346 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:20.131140 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:20.132102 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.132127 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.133844 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.133852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.136087 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.136103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.137312 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.137324 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.137724 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.138824 (+ 1100us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:20.138855 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:20.142127 (+ 3272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:20.145884 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:20.145899 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:20.145925 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:20.146749 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:20.147153 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:20.147161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:20.147167 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:20.147556 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:20.148200 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:20.148209 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:20.148585 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 13:00:20.148907 (+ 322us) client_negotiation.cc:240] Negotiation successful
0619 13:00:20.149577 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":591,"threads_started":1}
I20260619 13:00:20.179906 3340 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:40679
I20260619 13:00:20.179981 3474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:40679 every 8 connection(s)
I20260619 13:00:20.181396 3340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:20.183665 3475 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37575
I20260619 13:00:20.183907 3475 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:20.184532 3475 heartbeater.cc:511] Master 127.1.87.126:37575 requested a full tablet report, sending...
I20260619 13:00:20.185813 3127 ts_manager.cc:194] Registered new tserver with Master: 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679)
I20260619 13:00:20.186610 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.166587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:53973 (local address 127.1.87.126:37575)
0619 13:00:20.166832 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:20.166842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:20.167746 (+ 904us) server_negotiation.cc:407] Connection header received
0619 13:00:20.167993 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:20.168002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:20.168106 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:20.168246 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:20.170253 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.171768 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.174911 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.175925 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.176154 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.179108 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:20.179116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:20.179123 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:20.179182 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:20.182391 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.183300 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.183306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.183335 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.183507 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.185154 (+ 1647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.185158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.185160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.185476 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:20.185632 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:20.186019 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 13:00:20.186243 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 13:00:20.186672 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.166626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37575 (local address 127.1.87.66:53973)
0619 13:00:20.167582 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:20.167605 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:20.167817 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:20.168582 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:20.168588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:20.169523 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:20.170018 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.170030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.172051 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.172056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.174636 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.174665 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.176227 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.176238 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.176486 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.177507 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:20.177525 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:20.178830 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:20.182572 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:20.182577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:20.182580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:20.183191 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:20.183628 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:20.183635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:20.183652 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:20.183803 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:20.185608 (+ 1805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:20.185613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:20.185782 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 13:00:20.185929 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 13:00:20.186243 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":484,"threads_started":1}
I20260619 13:00:20.186628 3127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:56573
I20260619 13:00:20.189987 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 3340
I20260619 13:00:20.190155 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:20.190620 1373 external_mini_cluster.cc:1428] Setting key 5cd8e568afb675ad4dde70f8724b19f8
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:00:20.230279 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37575
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:45463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:20.563196 3479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:20.563515 3479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:20.563709 3479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:20.571163 3479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:20.571267 3479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:20.571466 3479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 13:00:20.571557 3479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:00:20.571691 3479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:20.581835 3479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:45463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.87.126:37575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:20.583123 3479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:20.584560 3479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:20.597795 3484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:20.597851 3487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:20.597878 3485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:20.598995 3479 server_base.cc:1034] running on GCE node
I20260619 13:00:20.599817 3479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:20.600769 3479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:20.601967 3479 hybrid_clock.cc:648] HybridClock initialized: now 1781874020601916 us; error 39 us; skew 500 ppm
I20260619 13:00:20.602289 3479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:20 dist-test-slave-tpsm krb5kdc[3083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:20.605325 3479 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:00:20.606844 3479 webserver.cc:460] Webserver started at http://127.1.87.67:37971/ using document root <none> and password file <none>
I20260619 13:00:20.607559 3479 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:20.607667 3479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:20.607928 3479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:20.610913 3479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "2c15a5c567d04fbd92a402b42f0a220f"
format_stamp: "Formatted at 2026-06-19 13:00:20 on dist-test-slave-tpsm"
server_key: "53ba83c5c79503ee17f901eba39ebfdf"
server_key_iv: "d021066a57acf1dc642840d25295b383"
server_key_version: "encryptionkey@0"
I20260619 13:00:20.611725 3479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "2c15a5c567d04fbd92a402b42f0a220f"
format_stamp: "Formatted at 2026-06-19 13:00:20 on dist-test-slave-tpsm"
server_key: "53ba83c5c79503ee17f901eba39ebfdf"
server_key_iv: "d021066a57acf1dc642840d25295b383"
server_key_version: "encryptionkey@0"
I20260619 13:00:20.617121 3479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:00:20.620586 3494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:20.621922 3479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:00:20.622100 3479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "2c15a5c567d04fbd92a402b42f0a220f"
format_stamp: "Formatted at 2026-06-19 13:00:20 on dist-test-slave-tpsm"
server_key: "53ba83c5c79503ee17f901eba39ebfdf"
server_key_iv: "d021066a57acf1dc642840d25295b383"
server_key_version: "encryptionkey@0"
I20260619 13:00:20.622310 3479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:20.642731 3479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:20.646934 3479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:20.647662 3479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:20.650851 3479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:20.651104 3479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:20.651253 3479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:20.651374 3479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:20.661041 3336 heartbeater.cc:503] Master 127.1.87.126:37575 was elected leader, sending a full tablet report...
Jun 19 13:00:20 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:20.678401 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.651695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:49125 (local address 127.1.87.126:37575)
0619 13:00:20.651889 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:20.651896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:20.653768 (+ 1872us) server_negotiation.cc:407] Connection header received
0619 13:00:20.656583 (+ 2815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:20.656589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:20.656692 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:20.656809 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:20.658436 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.659895 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.663667 (+ 3772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.664433 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.664595 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.669692 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:20.669702 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:20.669707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:20.669762 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:20.674170 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.675016 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.675025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.675031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.675216 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.675951 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.675958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.675963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.676389 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:20.676566 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:20.677510 (+ 944us) server_negotiation.cc:299] Negotiation successful
0619 13:00:20.677843 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:00:20.679219 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.652471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37575 (local address 127.1.87.67:49125)
0619 13:00:20.653607 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:20.653652 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:20.656237 (+ 2585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:20.657062 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:20.657070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:20.657614 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:20.658251 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.658268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.660107 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.660116 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.663100 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.663122 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.664759 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.664775 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.665201 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.666150 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:20.666186 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:20.669384 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:20.674383 (+ 4999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:20.674390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:20.674410 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:20.674893 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:20.675410 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:20.675418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:20.675428 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:20.675777 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:20.676538 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:20.676546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:20.677036 (+ 490us) client_negotiation.cc:769] Sending connection context
0619 13:00:20.677354 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 13:00:20.677798 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":449,"threads_started":1}
I20260619 13:00:20.711045 3479 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:36671
I20260619 13:00:20.711071 3613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:36671 every 8 connection(s)
I20260619 13:00:20.711580 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.692528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:38385 (local address 127.1.87.126:37575)
0619 13:00:20.692743 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:20.692752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:20.693599 (+ 847us) server_negotiation.cc:407] Connection header received
0619 13:00:20.693888 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:20.693895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:20.693987 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:20.694078 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:20.696029 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.697127 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.700343 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.701062 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.701178 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.704733 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:20.704742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:20.704747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:20.704800 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:20.708351 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.709262 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.709267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.709271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.709400 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.709779 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.709784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.709787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.710091 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:20.710247 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:20.710900 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 13:00:20.711132 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 13:00:20.711671 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.692474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37575 (local address 127.1.87.67:38385)
0619 13:00:20.693448 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:20.693477 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:20.693668 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:20.694353 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:20.694361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:20.695162 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:20.695783 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.695800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.697334 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.697343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.700129 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.700165 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.701282 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.701289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.701484 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.702657 (+ 1173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:20.702694 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:20.704475 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:20.708488 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:20.708494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:20.708498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:20.709110 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:20.709519 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:20.709524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:20.709527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:20.709662 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:20.710317 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:20.710324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:20.710611 (+ 287us) client_negotiation.cc:769] Sending connection context
0619 13:00:20.710803 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 13:00:20.711155 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":555,"threads_started":1}
I20260619 13:00:20.712745 3479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:20.715082 3614 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37575
I20260619 13:00:20.715356 3614 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:20.716004 3614 heartbeater.cc:511] Master 127.1.87.126:37575 requested a full tablet report, sending...
I20260619 13:00:20.717146 3127 ts_manager.cc:194] Registered new tserver with Master: 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671)
I20260619 13:00:20.717923 3127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:49125
I20260619 13:00:20.721015 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 3479
I20260619 13:00:20.721199 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:20.721612 1373 external_mini_cluster.cc:1428] Setting key 7990a9efedbf29c43dd32bc189b495f5
I20260619 13:00:20.724265 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:00:20 dist-test-slave-tpsm krb5kdc[3083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:00:20 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:20.771426 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.756411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.1.87.126:37575)
0619 13:00:20.756569 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:20.756575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:20.756938 (+ 363us) server_negotiation.cc:407] Connection header received
0619 13:00:20.757084 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:20.757088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:20.757147 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:20.757238 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:20.758600 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.759446 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.760692 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.761120 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.761228 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.765417 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:20.765422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:20.765425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:20.765459 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:20.768513 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.769418 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.769423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.769425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.769587 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:20.770016 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:20.770020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:20.770022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:20.770378 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:20.770585 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:20.770912 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 13:00:20.771071 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:00:20.775760 3127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:00:20.782977 3127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:20.806177 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.795675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36671 (local address 127.0.0.1:58588)
0619 13:00:20.797271 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:20.797299 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:20.797494 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:20.798214 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:20.798224 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:20.798255 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:20.798740 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.798752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.801682 (+ 2930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.801687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.803885 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.803899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.805149 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.805155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.805344 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.805396 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 13:00:20.805559 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 13:00:20.805756 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1380,"thread_start_us":2111,"threads_started":1}
I20260619 13:00:20.806403 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.796451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.1.87.65:40749)
0619 13:00:20.798627 (+ 2176us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:20.798635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:20.798663 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:00:20.798755 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:20.798762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:20.799012 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:20.799834 (+ 822us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:20.799989 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.801037 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.803977 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.805381 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.805501 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.805693 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:20.805931 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 13:00:20.806117 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2023,"thread_start_us":1888,"threads_started":1}
I20260619 13:00:20.807250 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.795889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.1.87.67:36671)
0619 13:00:20.797007 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:20.797022 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:20.797411 (+ 389us) server_negotiation.cc:407] Connection header received
0619 13:00:20.797703 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:20.797714 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:20.797993 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:20.799078 (+ 1085us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:20.799452 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.801466 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.804059 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.804961 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.805125 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.805366 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:20.806559 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0619 13:00:20.806748 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":729,"threads_started":1}
I20260619 13:00:20.808030 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.795850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40749 (local address 127.0.0.1:40856)
0619 13:00:20.798033 (+ 2183us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:20.798051 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:20.798157 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:20.799217 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:20.799222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:20.799243 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:20.799649 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.799663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.801158 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.801165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.803794 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.803815 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.805581 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.805587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.805747 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.805774 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 13:00:20.805868 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 13:00:20.806008 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2062,"mutex_wait_us":26,"thread_start_us":1083,"threads_started":1}
I20260619 13:00:20.809891 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.798821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40679 (local address 127.0.0.1:55680)
0619 13:00:20.802175 (+ 3354us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:20.802189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:20.802330 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:20.803546 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:20.803550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:20.803568 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:20.804007 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.804017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.805763 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.805767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.807769 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:20.807780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.809146 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.809153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:20.809354 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.809387 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:00:20.809473 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 13:00:20.809590 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3233,"mutex_wait_us":2,"thread_start_us":3081,"threads_started":1}
I20260619 13:00:20.810016 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:20.799343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.1.87.66:40679)
0619 13:00:20.801543 (+ 2200us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:20.801551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:20.802469 (+ 918us) server_negotiation.cc:407] Connection header received
0619 13:00:20.802567 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:20.802576 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:20.802795 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:20.802981 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:20.804255 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.805608 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.807913 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:20.808918 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:20.809086 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:20.809302 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:20.809559 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 13:00:20.809736 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2053,"thread_start_us":1927,"threads_started":1}
I20260619 13:00:20.811168 3545 tablet_service.cc:1467] Processing CreateTablet for tablet 187496fdeb5e4873b3229c8b6efc2b7a (DEFAULT_TABLE table=test-table [id=0e201ee4a07c41b48e57237834eb67b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:20.811187 3266 tablet_service.cc:1467] Processing CreateTablet for tablet 187496fdeb5e4873b3229c8b6efc2b7a (DEFAULT_TABLE table=test-table [id=0e201ee4a07c41b48e57237834eb67b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:20.812438 3545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187496fdeb5e4873b3229c8b6efc2b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:20.812505 3266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187496fdeb5e4873b3229c8b6efc2b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:20.813344 3405 tablet_service.cc:1467] Processing CreateTablet for tablet 187496fdeb5e4873b3229c8b6efc2b7a (DEFAULT_TABLE table=test-table [id=0e201ee4a07c41b48e57237834eb67b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:20.814560 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187496fdeb5e4873b3229c8b6efc2b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:20.823886 3633 tablet_bootstrap.cc:492] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: Bootstrap starting.
I20260619 13:00:20.826239 3634 tablet_bootstrap.cc:492] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: Bootstrap starting.
I20260619 13:00:20.828100 3633 tablet_bootstrap.cc:654] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:20.829526 3633 log.cc:824] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:20.829851 3635 tablet_bootstrap.cc:492] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: Bootstrap starting.
I20260619 13:00:20.829907 3634 tablet_bootstrap.cc:654] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:20.831286 3634 log.cc:824] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:20.832773 3635 tablet_bootstrap.cc:654] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:20.833698 3633 tablet_bootstrap.cc:492] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: No bootstrap required, opened a new log
I20260619 13:00:20.833900 3635 log.cc:824] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:20.833948 3633 ts_tablet_manager.cc:1397] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 13:00:20.834164 3634 tablet_bootstrap.cc:492] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: No bootstrap required, opened a new log
I20260619 13:00:20.834470 3634 ts_tablet_manager.cc:1397] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260619 13:00:20.836668 3635 tablet_bootstrap.cc:492] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: No bootstrap required, opened a new log
I20260619 13:00:20.836987 3635 ts_tablet_manager.cc:1397] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260619 13:00:20.841507 3633 raft_consensus.cc:348] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } }
I20260619 13:00:20.841876 3633 raft_consensus.cc:374] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:20.841969 3633 raft_consensus.cc:729] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c15a5c567d04fbd92a402b42f0a220f, State: Initialized, Role: FOLLOWER
I20260619 13:00:20.842562 3633 consensus_queue.cc:260] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } }
I20260619 13:00:20.843617 3614 heartbeater.cc:503] Master 127.1.87.126:37575 was elected leader, sending a full tablet report...
I20260619 13:00:20.844604 3634 raft_consensus.cc:348] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } }
I20260619 13:00:20.845000 3634 raft_consensus.cc:374] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:20.845080 3634 raft_consensus.cc:729] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb54da8bae14050a9d1f5c2d60d9e97, State: Initialized, Role: FOLLOWER
I20260619 13:00:20.845382 3633 ts_tablet_manager.cc:1428] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260619 13:00:20.845734 3634 consensus_queue.cc:260] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } }
I20260619 13:00:20.847385 3634 ts_tablet_manager.cc:1428] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 13:00:20.847123 3635 raft_consensus.cc:348] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } }
I20260619 13:00:20.847548 3635 raft_consensus.cc:374] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:20.847640 3635 raft_consensus.cc:729] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6796dd76bd9740ae844e6f681546cd0c, State: Initialized, Role: FOLLOWER
I20260619 13:00:20.848850 3635 consensus_queue.cc:260] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } }
I20260619 13:00:20.849942 3475 heartbeater.cc:503] Master 127.1.87.126:37575 was elected leader, sending a full tablet report...
I20260619 13:00:20.850502 3635 ts_tablet_manager.cc:1428] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260619 13:00:20.903916 3337 tablet.cc:2369] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:00:20.933992 3476 tablet.cc:2369] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:00:20.964780 3615 tablet.cc:2369] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:00:21.101480 3639 raft_consensus.cc:482] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:21.101832 3639 raft_consensus.cc:504] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } }
I20260619 13:00:21.103834 3639 leader_election.cc:288] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749), 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679)
I20260619 13:00:21.110276 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.104417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:54675 (local address 127.1.87.65:40749)
0619 13:00:21.104605 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.104611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.104631 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:00:21.104882 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.104887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.104980 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.105063 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:21.105650 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.106717 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.108594 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.109390 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.109543 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.109653 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.109909 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.110031 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:00:21.110349 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.104183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40749 (local address 127.1.87.67:54675)
0619 13:00:21.104535 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:21.104558 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:21.104683 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:21.105248 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:21.105253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:21.105273 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:21.105498 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.105507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.106867 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.106872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.108433 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.108445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.109565 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.109571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.109758 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.109782 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:00:21.109885 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 13:00:21.110020 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260619 13:00:21.111510 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187496fdeb5e4873b3229c8b6efc2b7a" candidate_uuid: "2c15a5c567d04fbd92a402b42f0a220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" is_pre_election: true
I20260619 13:00:21.112056 3286 raft_consensus.cc:2454] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c15a5c567d04fbd92a402b42f0a220f in term 0.
I20260619 13:00:21.112845 3498 leader_election.cc:302] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15a5c567d04fbd92a402b42f0a220f, eeb54da8bae14050a9d1f5c2d60d9e97; no voters:
I20260619 13:00:21.113296 3639 raft_consensus.cc:2781] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:21.113404 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.104411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40679 (local address 127.1.87.67:44405)
0619 13:00:21.105088 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:21.105104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:21.105259 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:21.105715 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:21.105721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:21.105748 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:21.106053 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.106063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.107944 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.107953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.110546 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.110563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.112575 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.112581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.112776 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.112810 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:00:21.112916 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:00:21.113054 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":662,"threads_started":1}
I20260619 13:00:21.113430 3639 raft_consensus.cc:482] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:21.113488 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.104841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:44405 (local address 127.1.87.66:40679)
0619 13:00:21.105068 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.105077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.105173 (+ 96us) server_negotiation.cc:407] Connection header received
0619 13:00:21.105476 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.105484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.105581 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.105705 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:21.106222 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.107726 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.110762 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.112308 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.112475 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.112621 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.112971 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.113154 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:00:21.113534 3639 raft_consensus.cc:3037] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:21.114676 3426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187496fdeb5e4873b3229c8b6efc2b7a" candidate_uuid: "2c15a5c567d04fbd92a402b42f0a220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6796dd76bd9740ae844e6f681546cd0c" is_pre_election: true
I20260619 13:00:21.115177 3426 raft_consensus.cc:2454] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c15a5c567d04fbd92a402b42f0a220f in term 0.
I20260619 13:00:21.116485 3639 raft_consensus.cc:504] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } }
I20260619 13:00:21.117081 3639 leader_election.cc:288] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [CANDIDATE]: Term 1 election: Requested vote from peers eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749), 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679)
I20260619 13:00:21.117483 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187496fdeb5e4873b3229c8b6efc2b7a" candidate_uuid: "2c15a5c567d04fbd92a402b42f0a220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97"
I20260619 13:00:21.117508 3426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187496fdeb5e4873b3229c8b6efc2b7a" candidate_uuid: "2c15a5c567d04fbd92a402b42f0a220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6796dd76bd9740ae844e6f681546cd0c"
I20260619 13:00:21.117690 3286 raft_consensus.cc:3037] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:21.117707 3426 raft_consensus.cc:3037] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:21.120049 3286 raft_consensus.cc:2454] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c15a5c567d04fbd92a402b42f0a220f in term 1.
I20260619 13:00:21.120214 3426 raft_consensus.cc:2454] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c15a5c567d04fbd92a402b42f0a220f in term 1.
I20260619 13:00:21.120471 3498 leader_election.cc:302] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15a5c567d04fbd92a402b42f0a220f, eeb54da8bae14050a9d1f5c2d60d9e97; no voters:
I20260619 13:00:21.120762 3639 raft_consensus.cc:2781] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:21.121172 3639 raft_consensus.cc:686] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 1 LEADER]: Becoming Leader. State: Replica: 2c15a5c567d04fbd92a402b42f0a220f, State: Running, Role: LEADER
I20260619 13:00:21.121718 3639 consensus_queue.cc:237] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } }
I20260619 13:00:21.126170 3128 catalog_manager.cc:5495] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67). New cstate: current_term: 1 leader_uuid: "2c15a5c567d04fbd92a402b42f0a220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } health_report { overall_health: UNKNOWN } } }
Jun 19 13:00:21 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874018, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:21.206993 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.193605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37575 (local address 127.0.0.1:54846)
0619 13:00:21.194134 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:21.194149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:21.194325 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:21.194644 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:21.194648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:21.195295 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:21.195613 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.195622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.196557 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.196564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.197491 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.197500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.198201 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.198206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.198381 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.199069 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:21.199087 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:21.201404 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:21.204696 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:21.204706 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:21.204715 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:21.205125 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:21.205471 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:21.205474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:21.205477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:21.205609 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:21.206082 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:21.206088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:21.206237 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 13:00:21.206389 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 13:00:21.206583 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":295,"threads_started":1}
I20260619 13:00:21.207050 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.193584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.1.87.126:37575)
0619 13:00:21.193763 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.193768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.194240 (+ 472us) server_negotiation.cc:407] Connection header received
0619 13:00:21.194448 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.194453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.194522 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.194621 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:21.195745 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.196437 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.197618 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.198027 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.198167 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.201606 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:21.201611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:21.201614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:21.201642 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:21.204559 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:21.205235 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:21.205239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:21.205241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:21.205373 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:21.205707 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:21.205710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:21.205712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:21.205957 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:21.206078 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.206580 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.206733 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:00:21.211956 3128 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:54846:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 13:00:21.212931 3128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:21.219539 3405 tablet_service.cc:1467] Processing CreateTablet for tablet a15828dcd6cf4c3ba5a65de0d1c4bb08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9217904c5d3142e4a65c11565eacef7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:00:21.219707 3545 tablet_service.cc:1467] Processing CreateTablet for tablet a15828dcd6cf4c3ba5a65de0d1c4bb08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9217904c5d3142e4a65c11565eacef7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:00:21.219571 3266 tablet_service.cc:1467] Processing CreateTablet for tablet a15828dcd6cf4c3ba5a65de0d1c4bb08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9217904c5d3142e4a65c11565eacef7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:00:21.220387 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a15828dcd6cf4c3ba5a65de0d1c4bb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:21.220647 3545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a15828dcd6cf4c3ba5a65de0d1c4bb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:21.220665 3266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a15828dcd6cf4c3ba5a65de0d1c4bb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:21.226228 3635 tablet_bootstrap.cc:492] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: Bootstrap starting.
I20260619 13:00:21.227746 3635 tablet_bootstrap.cc:654] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:21.228598 3634 tablet_bootstrap.cc:492] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: Bootstrap starting.
I20260619 13:00:21.228699 3633 tablet_bootstrap.cc:492] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: Bootstrap starting.
I20260619 13:00:21.230193 3634 tablet_bootstrap.cc:654] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:21.230352 3633 tablet_bootstrap.cc:654] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:21.230563 3635 tablet_bootstrap.cc:492] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: No bootstrap required, opened a new log
I20260619 13:00:21.230705 3635 ts_tablet_manager.cc:1397] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 13:00:21.231601 3635 raft_consensus.cc:348] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.231858 3635 raft_consensus.cc:374] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:21.231959 3635 raft_consensus.cc:729] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6796dd76bd9740ae844e6f681546cd0c, State: Initialized, Role: FOLLOWER
I20260619 13:00:21.232210 3635 consensus_queue.cc:260] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.232458 3634 tablet_bootstrap.cc:492] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: No bootstrap required, opened a new log
I20260619 13:00:21.232594 3634 ts_tablet_manager.cc:1397] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:00:21.232757 3635 ts_tablet_manager.cc:1428] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:00:21.232631 3641 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.233013 3641 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:21.233302 3634 raft_consensus.cc:348] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.233608 3634 raft_consensus.cc:374] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:21.233713 3634 raft_consensus.cc:729] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb54da8bae14050a9d1f5c2d60d9e97, State: Initialized, Role: FOLLOWER
I20260619 13:00:21.233948 3634 consensus_queue.cc:260] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.234602 3634 ts_tablet_manager.cc:1428] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:00:21.234432 3640 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.234823 3640 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:21.235492 3633 tablet_bootstrap.cc:492] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: No bootstrap required, opened a new log
I20260619 13:00:21.235659 3633 ts_tablet_manager.cc:1397] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20260619 13:00:21.236388 3633 raft_consensus.cc:348] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.236582 3633 raft_consensus.cc:374] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:21.236683 3633 raft_consensus.cc:729] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c15a5c567d04fbd92a402b42f0a220f, State: Initialized, Role: FOLLOWER
I20260619 13:00:21.236969 3633 consensus_queue.cc:260] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.237430 3639 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.237700 3633 ts_tablet_manager.cc:1428] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 13:00:21.237754 3639 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:21.278228 3639 raft_consensus.cc:482] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:21.278435 3639 raft_consensus.cc:504] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.278970 3639 leader_election.cc:288] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749), 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679)
I20260619 13:00:21.279628 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15828dcd6cf4c3ba5a65de0d1c4bb08" candidate_uuid: "2c15a5c567d04fbd92a402b42f0a220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" is_pre_election: true
I20260619 13:00:21.279628 3426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15828dcd6cf4c3ba5a65de0d1c4bb08" candidate_uuid: "2c15a5c567d04fbd92a402b42f0a220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6796dd76bd9740ae844e6f681546cd0c" is_pre_election: true
I20260619 13:00:21.279893 3426 raft_consensus.cc:2454] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c15a5c567d04fbd92a402b42f0a220f in term 0.
I20260619 13:00:21.279896 3286 raft_consensus.cc:2454] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c15a5c567d04fbd92a402b42f0a220f in term 0.
I20260619 13:00:21.280333 3498 leader_election.cc:302] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c15a5c567d04fbd92a402b42f0a220f, 6796dd76bd9740ae844e6f681546cd0c, eeb54da8bae14050a9d1f5c2d60d9e97; no voters:
I20260619 13:00:21.280692 3639 raft_consensus.cc:2781] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:21.280793 3639 raft_consensus.cc:482] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:21.280982 3639 raft_consensus.cc:3037] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:21.283758 3639 raft_consensus.cc:504] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.284614 3639 leader_election.cc:288] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [CANDIDATE]: Term 1 election: Requested vote from peers eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749), 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679)
I20260619 13:00:21.285022 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15828dcd6cf4c3ba5a65de0d1c4bb08" candidate_uuid: "2c15a5c567d04fbd92a402b42f0a220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97"
I20260619 13:00:21.285272 3286 raft_consensus.cc:3037] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:21.285254 3426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15828dcd6cf4c3ba5a65de0d1c4bb08" candidate_uuid: "2c15a5c567d04fbd92a402b42f0a220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6796dd76bd9740ae844e6f681546cd0c"
I20260619 13:00:21.285465 3426 raft_consensus.cc:3037] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:21.288288 3286 raft_consensus.cc:2454] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c15a5c567d04fbd92a402b42f0a220f in term 1.
I20260619 13:00:21.288293 3426 raft_consensus.cc:2454] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c15a5c567d04fbd92a402b42f0a220f in term 1.
I20260619 13:00:21.288717 3495 leader_election.cc:302] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15a5c567d04fbd92a402b42f0a220f, 6796dd76bd9740ae844e6f681546cd0c; no voters:
I20260619 13:00:21.289032 3639 raft_consensus.cc:2781] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:21.289407 3639 raft_consensus.cc:686] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [term 1 LEADER]: Becoming Leader. State: Replica: 2c15a5c567d04fbd92a402b42f0a220f, State: Running, Role: LEADER
I20260619 13:00:21.289664 3639 consensus_queue.cc:237] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.291357 3643 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: TxnStatusTablet state changed. Reason: New leader 2c15a5c567d04fbd92a402b42f0a220f. Latest consensus state: current_term: 1 leader_uuid: "2c15a5c567d04fbd92a402b42f0a220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.291663 3643 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:00:21.292294 3655 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 13:00:21.292560 3655 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 13:00:21.292618 3127 catalog_manager.cc:5495] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67). New cstate: current_term: 1 leader_uuid: "2c15a5c567d04fbd92a402b42f0a220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } health_report { overall_health: HEALTHY } } }
I20260619 13:00:21.320307 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.313674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.1.87.67:36671)
0619 13:00:21.314156 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.314167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.314181 (+ 14us) server_negotiation.cc:407] Connection header received
0619 13:00:21.314277 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.314283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.314399 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.314581 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:21.314950 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.315971 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.317238 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.317900 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.318074 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.318689 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.319266 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.319802 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.319900 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.319977 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":203,"threads_started":1}
I20260619 13:00:21.320307 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.313558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36671 (local address 127.0.0.1:58598)
0619 13:00:21.313796 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:21.313820 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:21.313967 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:21.314609 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:21.314614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:21.314627 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:00:21.314818 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.314827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.316103 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.316111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.317081 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.317091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.318086 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.318091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.318249 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.318427 (+ 178us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.319734 (+ 1307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.319761 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 13:00:21.319860 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 13:00:21.320020 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 13:00:21.329717 3426 raft_consensus.cc:1261] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Refusing update from remote peer 2c15a5c567d04fbd92a402b42f0a220f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:21.329717 3286 raft_consensus.cc:1261] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Refusing update from remote peer 2c15a5c567d04fbd92a402b42f0a220f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:21.330741 3643 consensus_queue.cc:1035] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:21.331054 3639 consensus_queue.cc:1035] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:21.342904 3640 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: TxnStatusTablet state changed. Reason: New leader 2c15a5c567d04fbd92a402b42f0a220f. Latest consensus state: current_term: 1 leader_uuid: "2c15a5c567d04fbd92a402b42f0a220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.343221 3640 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:21.345623 3639 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c15a5c567d04fbd92a402b42f0a220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.345849 3661 mvcc.cc:205] Tried to move back new op lower bound from 7298555991355715584 to 7298555991204978688. Current Snapshot: MvccSnapshot[applied={T|T < 7298555991355715584}]
I20260619 13:00:21.345916 3639 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:00:21.346702 3659 mvcc.cc:205] Tried to move back new op lower bound from 7298555991355715584 to 7298555991204978688. Current Snapshot: MvccSnapshot[applied={T|T < 7298555991355715584}]
I20260619 13:00:21.357726 3640 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c15a5c567d04fbd92a402b42f0a220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.357986 3640 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P eeb54da8bae14050a9d1f5c2d60d9e97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:21.358047 3641 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: TxnStatusTablet state changed. Reason: New leader 2c15a5c567d04fbd92a402b42f0a220f. Latest consensus state: current_term: 1 leader_uuid: "2c15a5c567d04fbd92a402b42f0a220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.358321 3641 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:21.360074 3643 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c15a5c567d04fbd92a402b42f0a220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.360338 3643 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 2c15a5c567d04fbd92a402b42f0a220f: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:00:21.363862 3660 mvcc.cc:205] Tried to move back new op lower bound from 7298555991355715584 to 7298555991204978688. Current Snapshot: MvccSnapshot[applied={T|T < 7298555991355715584}]
I20260619 13:00:21.364775 3641 tablet_replica.cc:439] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c15a5c567d04fbd92a402b42f0a220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } } }
I20260619 13:00:21.365082 3641 tablet_replica.cc:442] T a15828dcd6cf4c3ba5a65de0d1c4bb08 P 6796dd76bd9740ae844e6f681546cd0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:00:21.371495 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.363682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.1.87.67:36671)
0619 13:00:21.363861 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.363867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.366363 (+ 2496us) server_negotiation.cc:407] Connection header received
0619 13:00:21.366532 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.366538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.366622 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.366750 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:21.367140 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.368236 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.369802 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.370202 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.370350 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.370745 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.370850 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.370943 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.371093 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.371173 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:00:21.388640 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.382214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:42903 (local address 127.1.87.67:36671)
0619 13:00:21.382429 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.382435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.382787 (+ 352us) server_negotiation.cc:407] Connection header received
0619 13:00:21.382919 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.382924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.382998 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.383113 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:21.383686 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.384852 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.386458 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.387120 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.387263 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.387809 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.387960 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.388051 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.388168 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.388267 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 13:00:21.388659 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.382112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36671 (local address 127.1.87.67:42903)
0619 13:00:21.382681 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:21.382706 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:21.382815 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:21.383106 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:21.383109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:21.383118 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:00:21.383537 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.383548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.385015 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.385019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.386297 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.386318 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.387393 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.387398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.387521 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.387619 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.388057 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.388064 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:00:21.388126 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 13:00:21.388222 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":288,"threads_started":1}
I20260619 13:00:21.397806 3424 raft_consensus.cc:1261] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Refusing update from remote peer 2c15a5c567d04fbd92a402b42f0a220f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:21.397818 3286 raft_consensus.cc:1261] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Refusing update from remote peer 2c15a5c567d04fbd92a402b42f0a220f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:21.398316 3667 consensus_queue.cc:1035] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:21.398659 3639 consensus_queue.cc:1035] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:21.475643 3128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:21.475972 3128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:21.479490 3128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11: Sending DeleteTablet for 3 replicas of tablet 187496fdeb5e4873b3229c8b6efc2b7a
I20260619 13:00:21.480621 3543 tablet_service.cc:1514] Processing DeleteTablet for tablet 187496fdeb5e4873b3229c8b6efc2b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:21 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:58588
I20260619 13:00:21.480760 3266 tablet_service.cc:1514] Processing DeleteTablet for tablet 187496fdeb5e4873b3229c8b6efc2b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:21 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:40856
I20260619 13:00:21.480966 3405 tablet_service.cc:1514] Processing DeleteTablet for tablet 187496fdeb5e4873b3229c8b6efc2b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:21 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:55680
I20260619 13:00:21.482995 3128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:00:21.483793 3128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:21.490283 3679 tablet_replica.cc:330] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: stopping tablet replica
I20260619 13:00:21.490873 3679 raft_consensus.cc:2229] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:21.491017 3677 tablet_replica.cc:330] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: stopping tablet replica
I20260619 13:00:21.491417 3679 raft_consensus.cc:2258] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:21.491432 3677 raft_consensus.cc:2229] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:21.493355 3679 ts_tablet_manager.cc:1905] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:21.491786 3677 raft_consensus.cc:2258] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:21.495941 3678 tablet_replica.cc:330] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: stopping tablet replica
I20260619 13:00:21.495934 3265 tablet_service.cc:1467] Processing CreateTablet for tablet f9ae6aa3345040dfb2bf7a5afac17d7a (DEFAULT_TABLE table=test-table [id=a1ab50b5dea743838e5cb8b64ba38f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:21.496408 3265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae6aa3345040dfb2bf7a5afac17d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:21.496376 3678 raft_consensus.cc:2229] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:00:21.497028 3678 raft_consensus.cc:2258] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:21.499258 3678 ts_tablet_manager.cc:1905] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:21.499512 3545 tablet_service.cc:1467] Processing CreateTablet for tablet f9ae6aa3345040dfb2bf7a5afac17d7a (DEFAULT_TABLE table=test-table [id=a1ab50b5dea743838e5cb8b64ba38f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:21.500324 3677 ts_tablet_manager.cc:1905] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:21.500576 3405 tablet_service.cc:1467] Processing CreateTablet for tablet f9ae6aa3345040dfb2bf7a5afac17d7a (DEFAULT_TABLE table=test-table [id=a1ab50b5dea743838e5cb8b64ba38f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:21.500980 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae6aa3345040dfb2bf7a5afac17d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:21.503545 3679 ts_tablet_manager.cc:1918] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:00:21.507043 3545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae6aa3345040dfb2bf7a5afac17d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:21.507248 3679 log.cc:1196] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/187496fdeb5e4873b3229c8b6efc2b7a
I20260619 13:00:21.507241 3678 ts_tablet_manager.cc:1918] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:00:21.507417 3678 log.cc:1196] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/187496fdeb5e4873b3229c8b6efc2b7a
I20260619 13:00:21.507946 3679 ts_tablet_manager.cc:1939] T 187496fdeb5e4873b3229c8b6efc2b7a P 6796dd76bd9740ae844e6f681546cd0c: Deleting consensus metadata
I20260619 13:00:21.507994 3678 ts_tablet_manager.cc:1939] T 187496fdeb5e4873b3229c8b6efc2b7a P 2c15a5c567d04fbd92a402b42f0a220f: Deleting consensus metadata
I20260619 13:00:21.509757 3113 catalog_manager.cc:4841] TS 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671): tablet 187496fdeb5e4873b3229c8b6efc2b7a (table test-table [id=0e201ee4a07c41b48e57237834eb67b8]) successfully deleted
I20260619 13:00:21.510039 3112 catalog_manager.cc:4841] TS 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679): tablet 187496fdeb5e4873b3229c8b6efc2b7a (table test-table [id=0e201ee4a07c41b48e57237834eb67b8]) successfully deleted
I20260619 13:00:21.513250 3633 tablet_bootstrap.cc:492] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: Bootstrap starting.
I20260619 13:00:21.513499 3677 ts_tablet_manager.cc:1918] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:00:21.513706 3677 log.cc:1196] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/187496fdeb5e4873b3229c8b6efc2b7a
I20260619 13:00:21.514261 3677 ts_tablet_manager.cc:1939] T 187496fdeb5e4873b3229c8b6efc2b7a P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting consensus metadata
I20260619 13:00:21.514902 3633 tablet_bootstrap.cc:654] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:21.515677 3635 tablet_bootstrap.cc:492] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: Bootstrap starting.
I20260619 13:00:21.516978 3634 tablet_bootstrap.cc:492] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: Bootstrap starting.
I20260619 13:00:21.517218 3635 tablet_bootstrap.cc:654] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:21.518716 3634 tablet_bootstrap.cc:654] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:21.522539 3115 catalog_manager.cc:4841] TS eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749): tablet 187496fdeb5e4873b3229c8b6efc2b7a (table test-table [id=0e201ee4a07c41b48e57237834eb67b8]) successfully deleted
I20260619 13:00:21.524719 3635 tablet_bootstrap.cc:492] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: No bootstrap required, opened a new log
I20260619 13:00:21.524760 3634 tablet_bootstrap.cc:492] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: No bootstrap required, opened a new log
I20260619 13:00:21.524879 3635 ts_tablet_manager.cc:1397] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 13:00:21.524897 3634 ts_tablet_manager.cc:1397] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260619 13:00:21.525085 3633 tablet_bootstrap.cc:492] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: No bootstrap required, opened a new log
I20260619 13:00:21.525246 3633 ts_tablet_manager.cc:1397] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260619 13:00:21.525576 3634 raft_consensus.cc:348] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.525812 3634 raft_consensus.cc:374] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:21.525704 3635 raft_consensus.cc:348] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.525879 3634 raft_consensus.cc:729] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb54da8bae14050a9d1f5c2d60d9e97, State: Initialized, Role: FOLLOWER
I20260619 13:00:21.525931 3635 raft_consensus.cc:374] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:21.526005 3635 raft_consensus.cc:729] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6796dd76bd9740ae844e6f681546cd0c, State: Initialized, Role: FOLLOWER
I20260619 13:00:21.526037 3633 raft_consensus.cc:348] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.526297 3633 raft_consensus.cc:374] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:21.526381 3633 raft_consensus.cc:729] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c15a5c567d04fbd92a402b42f0a220f, State: Initialized, Role: FOLLOWER
I20260619 13:00:21.526237 3635 consensus_queue.cc:260] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.526424 3634 consensus_queue.cc:260] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.526798 3635 ts_tablet_manager.cc:1428] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:00:21.526657 3633 consensus_queue.cc:260] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.527244 3633 ts_tablet_manager.cc:1428] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 13:00:21.529143 3634 ts_tablet_manager.cc:1428] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 13:00:21.715458 3641 raft_consensus.cc:482] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:21.715672 3641 raft_consensus.cc:504] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.717216 3641 leader_election.cc:288] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749), 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671)
I20260619 13:00:21.733299 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.718040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:60907 (local address 127.1.87.67:36671)
0619 13:00:21.718244 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.718252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.725510 (+ 7258us) server_negotiation.cc:407] Connection header received
0619 13:00:21.725771 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.725777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.725855 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.725940 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:21.726697 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.727860 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.730502 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.732109 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.732252 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.732353 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.732867 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.733000 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:00:21.734485 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.717611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36671 (local address 127.1.87.66:60907)
0619 13:00:21.725368 (+ 7757us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:21.725397 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:21.725603 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:21.726111 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:21.726117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:21.726145 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:21.726521 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.726532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.728051 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.728056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.730327 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.730341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.732487 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.732492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.732689 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.732721 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:00:21.733766 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0619 13:00:21.733910 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7640,"thread_start_us":16478,"threads_started":1}
I20260619 13:00:21.735430 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae6aa3345040dfb2bf7a5afac17d7a" candidate_uuid: "6796dd76bd9740ae844e6f681546cd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15a5c567d04fbd92a402b42f0a220f" is_pre_election: true
I20260619 13:00:21.735831 3569 raft_consensus.cc:2454] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6796dd76bd9740ae844e6f681546cd0c in term 0.
I20260619 13:00:21.736512 3357 leader_election.cc:302] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15a5c567d04fbd92a402b42f0a220f, 6796dd76bd9740ae844e6f681546cd0c; no voters:
I20260619 13:00:21.736974 3641 raft_consensus.cc:2781] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:21.737080 3641 raft_consensus.cc:482] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:21.737149 3641 raft_consensus.cc:3037] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:21.739261 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.717773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40749 (local address 127.1.87.66:49725)
0619 13:00:21.731880 (+ 14107us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:21.731900 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:21.732632 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:21.733085 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:21.733090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:21.733130 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:21.733438 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.733448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.734871 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.734876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.736751 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.736764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.738472 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.738479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.738663 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.738696 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:00:21.738783 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 13:00:21.738916 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13984,"thread_start_us":7365,"threads_started":1}
I20260619 13:00:21.740065 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.718174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:49725 (local address 127.1.87.65:40749)
0619 13:00:21.732445 (+ 14271us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.732451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.732471 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:00:21.732793 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.732798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.732864 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.732946 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:21.733622 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.734739 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.736928 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.738107 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.738228 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.738324 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.739517 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.739753 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14151,"thread_start_us":14304,"threads_started":1}
I20260619 13:00:21.740592 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae6aa3345040dfb2bf7a5afac17d7a" candidate_uuid: "6796dd76bd9740ae844e6f681546cd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" is_pre_election: true
I20260619 13:00:21.740834 3286 raft_consensus.cc:2454] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6796dd76bd9740ae844e6f681546cd0c in term 0.
I20260619 13:00:21.740919 3641 raft_consensus.cc:504] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.741528 3641 leader_election.cc:288] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [CANDIDATE]: Term 1 election: Requested vote from peers eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749), 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671)
I20260619 13:00:21.741968 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae6aa3345040dfb2bf7a5afac17d7a" candidate_uuid: "6796dd76bd9740ae844e6f681546cd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97"
I20260619 13:00:21.742107 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae6aa3345040dfb2bf7a5afac17d7a" candidate_uuid: "6796dd76bd9740ae844e6f681546cd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15a5c567d04fbd92a402b42f0a220f"
I20260619 13:00:21.742165 3286 raft_consensus.cc:3037] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:21.742302 3569 raft_consensus.cc:3037] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:21.745115 3286 raft_consensus.cc:2454] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6796dd76bd9740ae844e6f681546cd0c in term 1.
I20260619 13:00:21.745546 3359 leader_election.cc:302] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6796dd76bd9740ae844e6f681546cd0c, eeb54da8bae14050a9d1f5c2d60d9e97; no voters:
I20260619 13:00:21.745590 3569 raft_consensus.cc:2454] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6796dd76bd9740ae844e6f681546cd0c in term 1.
I20260619 13:00:21.745865 3641 raft_consensus.cc:2781] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:21.746356 3641 raft_consensus.cc:686] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 1 LEADER]: Becoming Leader. State: Replica: 6796dd76bd9740ae844e6f681546cd0c, State: Running, Role: LEADER
I20260619 13:00:21.746747 3641 consensus_queue.cc:237] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:21.754817 3126 catalog_manager.cc:5495] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66). New cstate: current_term: 1 leader_uuid: "6796dd76bd9740ae844e6f681546cd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } health_report { overall_health: UNKNOWN } } }
I20260619 13:00:21.779596 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.772950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.1.87.66:40679)
0619 13:00:21.773498 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.773502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.773597 (+ 95us) server_negotiation.cc:407] Connection header received
0619 13:00:21.773808 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.773811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.773891 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.774032 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:21.774485 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.775247 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.777206 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.777707 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.777858 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.778291 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.778930 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.779077 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.779165 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.779246 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":305,"threads_started":1}
I20260619 13:00:21.795256 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.789866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:60747 (local address 127.1.87.67:36671)
0619 13:00:21.790083 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.790090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.790399 (+ 309us) server_negotiation.cc:407] Connection header received
0619 13:00:21.790630 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.790634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.790703 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.790804 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:21.791275 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.792047 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.793399 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.793887 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.794046 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.794458 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.794564 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.794684 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.794834 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.794914 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 13:00:21.795315 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.789759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36671 (local address 127.1.87.66:60747)
0619 13:00:21.790296 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:21.790312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:21.790475 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:21.790828 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:21.790832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:21.790843 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:00:21.791115 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.791122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.792185 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.792190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.793246 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.793257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.794072 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.794078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.794222 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.794276 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.794708 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.794724 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 13:00:21.794813 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 13:00:21.794944 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":254,"threads_started":1}
I20260619 13:00:21.803957 3286 raft_consensus.cc:1261] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Refusing update from remote peer 6796dd76bd9740ae844e6f681546cd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:21.804172 3569 raft_consensus.cc:1261] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Refusing update from remote peer 6796dd76bd9740ae844e6f681546cd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:21.804603 3686 consensus_queue.cc:1035] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:21.804962 3641 consensus_queue.cc:1035] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:21.834332 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.827570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40679 (local address 127.1.87.67:41857)
0619 13:00:21.827957 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:21.827976 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:21.828087 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:21.828607 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:21.828611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:21.828627 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:00:21.828828 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.828840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.830220 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.830225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.832163 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:21.832177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.833190 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.833195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:21.833370 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.833408 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.833779 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.833792 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 13:00:21.833869 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 13:00:21.834018 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260619 13:00:21.834909 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:21.827914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:41857 (local address 127.1.87.66:40679)
0619 13:00:21.828062 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:21.828068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:21.828085 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:00:21.828262 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:21.828267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:21.828352 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:21.828429 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:21.828987 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.830075 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.832471 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:21.832966 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:21.833080 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:21.833561 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:21.833683 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:21.833778 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:21.834659 (+ 881us) server_negotiation.cc:299] Negotiation successful
0619 13:00:21.834720 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 13:00:21.835830 3660 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 13:00:23.430919 3126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:23.431143 3126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:23.435406 3126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11: Sending DeleteTablet for 3 replicas of tablet f9ae6aa3345040dfb2bf7a5afac17d7a
I20260619 13:00:23.436189 3265 tablet_service.cc:1514] Processing DeleteTablet for tablet f9ae6aa3345040dfb2bf7a5afac17d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:23 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:40856
I20260619 13:00:23.436278 3545 tablet_service.cc:1514] Processing DeleteTablet for tablet f9ae6aa3345040dfb2bf7a5afac17d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:23 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:58588
I20260619 13:00:23.436280 3405 tablet_service.cc:1514] Processing DeleteTablet for tablet f9ae6aa3345040dfb2bf7a5afac17d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:23 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:55680
I20260619 13:00:23.447705 3720 tablet_replica.cc:330] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: stopping tablet replica
I20260619 13:00:23.448161 3720 raft_consensus.cc:2229] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:23.448559 3720 raft_consensus.cc:2258] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:23.449851 3720 ts_tablet_manager.cc:1905] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:23.450927 3721 tablet_replica.cc:330] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: stopping tablet replica
I20260619 13:00:23.451071 3719 tablet_replica.cc:330] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: stopping tablet replica
I20260619 13:00:23.451229 3721 raft_consensus.cc:2229] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:00:23.451285 3719 raft_consensus.cc:2229] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:23.451571 3719 raft_consensus.cc:2258] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:23.451956 3721 raft_consensus.cc:2258] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:23.452545 3719 ts_tablet_manager.cc:1905] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:23.452831 3721 ts_tablet_manager.cc:1905] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:23.457998 3720 ts_tablet_manager.cc:1918] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:00:23.458206 3720 log.cc:1196] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/f9ae6aa3345040dfb2bf7a5afac17d7a
I20260619 13:00:23.458767 3720 ts_tablet_manager.cc:1939] T f9ae6aa3345040dfb2bf7a5afac17d7a P 2c15a5c567d04fbd92a402b42f0a220f: Deleting consensus metadata
I20260619 13:00:23.459465 3719 ts_tablet_manager.cc:1918] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:00:23.459482 3721 ts_tablet_manager.cc:1918] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:00:23.459605 3721 log.cc:1196] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/f9ae6aa3345040dfb2bf7a5afac17d7a
I20260619 13:00:23.459650 3719 log.cc:1196] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/f9ae6aa3345040dfb2bf7a5afac17d7a
I20260619 13:00:23.460106 3721 ts_tablet_manager.cc:1939] T f9ae6aa3345040dfb2bf7a5afac17d7a P 6796dd76bd9740ae844e6f681546cd0c: Deleting consensus metadata
I20260619 13:00:23.460122 3719 ts_tablet_manager.cc:1939] T f9ae6aa3345040dfb2bf7a5afac17d7a P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting consensus metadata
I20260619 13:00:23.460208 3113 catalog_manager.cc:4841] TS 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671): tablet f9ae6aa3345040dfb2bf7a5afac17d7a (table test-table [id=a1ab50b5dea743838e5cb8b64ba38f46]) successfully deleted
I20260619 13:00:23.461164 3115 catalog_manager.cc:4841] TS eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749): tablet f9ae6aa3345040dfb2bf7a5afac17d7a (table test-table [id=a1ab50b5dea743838e5cb8b64ba38f46]) successfully deleted
I20260619 13:00:23.461212 3112 catalog_manager.cc:4841] TS 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679): tablet f9ae6aa3345040dfb2bf7a5afac17d7a (table test-table [id=a1ab50b5dea743838e5cb8b64ba38f46]) successfully deleted
Jun 19 13:00:23 dist-test-slave-tpsm krb5kdc[3083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:00:23 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:23.507115 3729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.486264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.1.87.126:37575)
0619 13:00:23.486828 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:23.486833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:23.486896 (+ 63us) server_negotiation.cc:407] Connection header received
0619 13:00:23.487077 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:23.487081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:23.487152 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:23.487254 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:23.488783 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.489556 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.491431 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.491842 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.491938 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.500484 (+ 8546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:23.500491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:23.500495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:23.500533 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:23.503941 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:23.504872 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:23.504882 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:23.504885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:23.505056 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:23.505563 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:23.505568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:23.505572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:23.505871 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:23.506139 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:23.506465 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 13:00:23.506706 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":290,"threads_started":1}
I20260619 13:00:23.510717 3126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:00:23.511366 3126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:23.518276 3405 tablet_service.cc:1467] Processing CreateTablet for tablet 4eaac5c87eca437f92314e49856515eb (DEFAULT_TABLE table=test-table [id=41363f70615247dc86fad068571376a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:23.518592 3545 tablet_service.cc:1467] Processing CreateTablet for tablet 4eaac5c87eca437f92314e49856515eb (DEFAULT_TABLE table=test-table [id=41363f70615247dc86fad068571376a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:23.518739 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eaac5c87eca437f92314e49856515eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:23.518922 3265 tablet_service.cc:1467] Processing CreateTablet for tablet 4eaac5c87eca437f92314e49856515eb (DEFAULT_TABLE table=test-table [id=41363f70615247dc86fad068571376a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:23.519032 3545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eaac5c87eca437f92314e49856515eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:23.519354 3265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eaac5c87eca437f92314e49856515eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:23.525370 3730 tablet_bootstrap.cc:492] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: Bootstrap starting.
I20260619 13:00:23.526993 3730 tablet_bootstrap.cc:654] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:23.527599 3732 tablet_bootstrap.cc:492] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: Bootstrap starting.
I20260619 13:00:23.529392 3732 tablet_bootstrap.cc:654] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:23.530273 3731 tablet_bootstrap.cc:492] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: Bootstrap starting.
I20260619 13:00:23.530979 3730 tablet_bootstrap.cc:492] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: No bootstrap required, opened a new log
I20260619 13:00:23.531145 3730 ts_tablet_manager.cc:1397] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 13:00:23.531845 3732 tablet_bootstrap.cc:492] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: No bootstrap required, opened a new log
I20260619 13:00:23.531991 3732 ts_tablet_manager.cc:1397] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:00:23.532001 3730 raft_consensus.cc:348] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.532284 3730 raft_consensus.cc:374] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:23.532424 3730 raft_consensus.cc:729] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c15a5c567d04fbd92a402b42f0a220f, State: Initialized, Role: FOLLOWER
I20260619 13:00:23.532742 3731 tablet_bootstrap.cc:654] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:23.532675 3730 consensus_queue.cc:260] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.533141 3730 ts_tablet_manager.cc:1428] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 13:00:23.532811 3732 raft_consensus.cc:348] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.533607 3732 raft_consensus.cc:374] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:23.533752 3732 raft_consensus.cc:729] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb54da8bae14050a9d1f5c2d60d9e97, State: Initialized, Role: FOLLOWER
I20260619 13:00:23.533994 3732 consensus_queue.cc:260] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.535025 3732 ts_tablet_manager.cc:1428] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 13:00:23.542037 3731 tablet_bootstrap.cc:492] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: No bootstrap required, opened a new log
I20260619 13:00:23.542240 3731 ts_tablet_manager.cc:1397] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260619 13:00:23.543057 3731 raft_consensus.cc:348] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.543304 3731 raft_consensus.cc:374] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:23.543419 3731 raft_consensus.cc:729] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6796dd76bd9740ae844e6f681546cd0c, State: Initialized, Role: FOLLOWER
I20260619 13:00:23.543655 3731 consensus_queue.cc:260] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.544124 3731 ts_tablet_manager.cc:1428] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:00:23.561033 3709 raft_consensus.cc:482] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:23.561282 3709 raft_consensus.cc:504] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.562012 3709 leader_election.cc:288] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749), 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671)
I20260619 13:00:23.562520 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eaac5c87eca437f92314e49856515eb" candidate_uuid: "6796dd76bd9740ae844e6f681546cd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15a5c567d04fbd92a402b42f0a220f" is_pre_election: true
I20260619 13:00:23.562552 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eaac5c87eca437f92314e49856515eb" candidate_uuid: "6796dd76bd9740ae844e6f681546cd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" is_pre_election: true
I20260619 13:00:23.562762 3569 raft_consensus.cc:2454] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6796dd76bd9740ae844e6f681546cd0c in term 0.
I20260619 13:00:23.562791 3286 raft_consensus.cc:2454] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6796dd76bd9740ae844e6f681546cd0c in term 0.
I20260619 13:00:23.563243 3357 leader_election.cc:302] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15a5c567d04fbd92a402b42f0a220f, 6796dd76bd9740ae844e6f681546cd0c; no voters:
I20260619 13:00:23.563511 3709 raft_consensus.cc:2781] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:23.563649 3709 raft_consensus.cc:482] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:23.563743 3709 raft_consensus.cc:3037] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:23.565943 3709 raft_consensus.cc:504] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.566490 3709 leader_election.cc:288] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [CANDIDATE]: Term 1 election: Requested vote from peers eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749), 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671)
I20260619 13:00:23.566954 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eaac5c87eca437f92314e49856515eb" candidate_uuid: "6796dd76bd9740ae844e6f681546cd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97"
I20260619 13:00:23.566955 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eaac5c87eca437f92314e49856515eb" candidate_uuid: "6796dd76bd9740ae844e6f681546cd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15a5c567d04fbd92a402b42f0a220f"
I20260619 13:00:23.567159 3569 raft_consensus.cc:3037] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:23.567160 3286 raft_consensus.cc:3037] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:23.582214 3569 raft_consensus.cc:2454] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6796dd76bd9740ae844e6f681546cd0c in term 1.
I20260619 13:00:23.582206 3286 raft_consensus.cc:2454] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6796dd76bd9740ae844e6f681546cd0c in term 1.
I20260619 13:00:23.582808 3359 leader_election.cc:302] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6796dd76bd9740ae844e6f681546cd0c, eeb54da8bae14050a9d1f5c2d60d9e97; no voters:
I20260619 13:00:23.583192 3709 raft_consensus.cc:2781] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:23.583424 3709 raft_consensus.cc:686] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 1 LEADER]: Becoming Leader. State: Replica: 6796dd76bd9740ae844e6f681546cd0c, State: Running, Role: LEADER
I20260619 13:00:23.583689 3709 consensus_queue.cc:237] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.586335 3126 catalog_manager.cc:5495] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66). New cstate: current_term: 1 leader_uuid: "6796dd76bd9740ae844e6f681546cd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } health_report { overall_health: UNKNOWN } } }
I20260619 13:00:23.616921 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.611482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.1.87.66:40679)
0619 13:00:23.611994 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:23.611998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:23.612016 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:00:23.612066 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:23.612070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:23.612126 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:23.612214 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:23.612825 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.613631 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.614889 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.615386 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.615491 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.616086 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:23.616199 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:23.616276 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:23.616545 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 13:00:23.616645 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":296,"threads_started":1}
I20260619 13:00:23.626196 3286 raft_consensus.cc:1261] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Refusing update from remote peer 6796dd76bd9740ae844e6f681546cd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:23.626309 3569 raft_consensus.cc:1261] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Refusing update from remote peer 6796dd76bd9740ae844e6f681546cd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:23.626793 3709 consensus_queue.cc:1035] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:23.627069 3697 consensus_queue.cc:1035] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:23.670709 3128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:23.670946 3128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:23.673844 3128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11: Sending DeleteTablet for 3 replicas of tablet 4eaac5c87eca437f92314e49856515eb
I20260619 13:00:23.674538 3265 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eaac5c87eca437f92314e49856515eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:23 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:40856
I20260619 13:00:23.674679 3545 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eaac5c87eca437f92314e49856515eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:23 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:58588
I20260619 13:00:23.674808 3719 tablet_replica.cc:330] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: stopping tablet replica
I20260619 13:00:23.674922 3405 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eaac5c87eca437f92314e49856515eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:23 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:55680
I20260619 13:00:23.675011 3719 raft_consensus.cc:2229] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:23.675199 3721 tablet_replica.cc:330] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: stopping tablet replica
I20260619 13:00:23.675220 3720 tablet_replica.cc:330] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: stopping tablet replica
I20260619 13:00:23.675227 3719 raft_consensus.cc:2258] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:23.675388 3721 raft_consensus.cc:2229] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:00:23.675455 3720 raft_consensus.cc:2229] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:23.675626 3720 raft_consensus.cc:2258] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:23.675717 3721 raft_consensus.cc:2258] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:23.676589 3721 ts_tablet_manager.cc:1905] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:23.676725 3720 ts_tablet_manager.cc:1905] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:23.677205 3719 ts_tablet_manager.cc:1905] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:23.680994 3128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:00:23.681821 3128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:23.688822 3545 tablet_service.cc:1467] Processing CreateTablet for tablet 1a89b2bdecbf49a58567dee13218e384 (DEFAULT_TABLE table=test-table [id=cb129f7d3f4940a19d373e3952f2f842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:23.689000 3405 tablet_service.cc:1467] Processing CreateTablet for tablet 1a89b2bdecbf49a58567dee13218e384 (DEFAULT_TABLE table=test-table [id=cb129f7d3f4940a19d373e3952f2f842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:23.689850 3265 tablet_service.cc:1467] Processing CreateTablet for tablet 1a89b2bdecbf49a58567dee13218e384 (DEFAULT_TABLE table=test-table [id=cb129f7d3f4940a19d373e3952f2f842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:23.699774 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a89b2bdecbf49a58567dee13218e384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:23.700191 3545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a89b2bdecbf49a58567dee13218e384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:23.704530 3265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a89b2bdecbf49a58567dee13218e384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:23.709168 3731 tablet_bootstrap.cc:492] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: Bootstrap starting.
I20260619 13:00:23.710178 3732 tablet_bootstrap.cc:492] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97: Bootstrap starting.
I20260619 13:00:23.710925 3731 tablet_bootstrap.cc:654] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:23.711447 3732 tablet_bootstrap.cc:654] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:23.713018 3731 tablet_bootstrap.cc:492] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: No bootstrap required, opened a new log
I20260619 13:00:23.713231 3731 ts_tablet_manager.cc:1397] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260619 13:00:23.713923 3731 raft_consensus.cc:348] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.714172 3731 raft_consensus.cc:374] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:23.714280 3731 raft_consensus.cc:729] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6796dd76bd9740ae844e6f681546cd0c, State: Initialized, Role: FOLLOWER
I20260619 13:00:23.714335 3732 tablet_bootstrap.cc:492] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97: No bootstrap required, opened a new log
I20260619 13:00:23.714509 3732 ts_tablet_manager.cc:1397] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260619 13:00:23.714571 3731 consensus_queue.cc:260] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.715194 3732 raft_consensus.cc:348] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.715448 3732 raft_consensus.cc:374] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:23.715538 3732 raft_consensus.cc:729] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb54da8bae14050a9d1f5c2d60d9e97, State: Initialized, Role: FOLLOWER
I20260619 13:00:23.715782 3732 consensus_queue.cc:260] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.715925 3731 ts_tablet_manager.cc:1428] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 13:00:23.716385 3732 ts_tablet_manager.cc:1428] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:00:23.718609 3720 ts_tablet_manager.cc:1918] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:00:23.718763 3720 log.cc:1196] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/4eaac5c87eca437f92314e49856515eb
I20260619 13:00:23.719184 3720 ts_tablet_manager.cc:1939] T 4eaac5c87eca437f92314e49856515eb P 2c15a5c567d04fbd92a402b42f0a220f: Deleting consensus metadata
I20260619 13:00:23.722735 3730 tablet_bootstrap.cc:492] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: Bootstrap starting.
I20260619 13:00:23.724375 3730 tablet_bootstrap.cc:654] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:23.726194 3730 tablet_bootstrap.cc:492] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: No bootstrap required, opened a new log
I20260619 13:00:23.726361 3730 ts_tablet_manager.cc:1397] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 13:00:23.726975 3730 raft_consensus.cc:348] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.727205 3730 raft_consensus.cc:374] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:23.727298 3730 raft_consensus.cc:729] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c15a5c567d04fbd92a402b42f0a220f, State: Initialized, Role: FOLLOWER
I20260619 13:00:23.727569 3730 consensus_queue.cc:260] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.728022 3730 ts_tablet_manager.cc:1428] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 13:00:23.770684 3113 catalog_manager.cc:4841] TS 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671): tablet 4eaac5c87eca437f92314e49856515eb (table test-table [id=41363f70615247dc86fad068571376a5]) successfully deleted
I20260619 13:00:23.799865 3721 ts_tablet_manager.cc:1918] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:00:23.799866 3719 ts_tablet_manager.cc:1918] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:00:23.800058 3721 log.cc:1196] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/4eaac5c87eca437f92314e49856515eb
I20260619 13:00:23.800057 3719 log.cc:1196] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/4eaac5c87eca437f92314e49856515eb
I20260619 13:00:23.800498 3719 ts_tablet_manager.cc:1939] T 4eaac5c87eca437f92314e49856515eb P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting consensus metadata
I20260619 13:00:23.800580 3721 ts_tablet_manager.cc:1939] T 4eaac5c87eca437f92314e49856515eb P 6796dd76bd9740ae844e6f681546cd0c: Deleting consensus metadata
I20260619 13:00:23.802742 3112 catalog_manager.cc:4841] TS 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679): tablet 4eaac5c87eca437f92314e49856515eb (table test-table [id=41363f70615247dc86fad068571376a5]) successfully deleted
I20260619 13:00:23.802709 3115 catalog_manager.cc:4841] TS eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749): tablet 4eaac5c87eca437f92314e49856515eb (table test-table [id=41363f70615247dc86fad068571376a5]) successfully deleted
I20260619 13:00:23.836424 3736 raft_consensus.cc:482] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:23.836668 3736 raft_consensus.cc:504] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.838296 3736 leader_election.cc:288] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679), 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671)
I20260619 13:00:23.847499 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.838855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:54923 (local address 127.1.87.66:40679)
0619 13:00:23.839092 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:23.839101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:23.839737 (+ 636us) server_negotiation.cc:407] Connection header received
0619 13:00:23.840041 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:23.840050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:23.840135 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:23.840183 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:23.841021 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.842181 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.845164 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.846368 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.846507 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.846607 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:23.847076 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 13:00:23.847205 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:00:23.847548 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.838798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:42427 (local address 127.1.87.67:36671)
0619 13:00:23.840226 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:23.840231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:23.840252 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:00:23.840305 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:23.840308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:23.840363 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:23.840446 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:23.841142 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.842237 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.845192 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.846388 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.846507 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.846612 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:23.847073 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 13:00:23.847205 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1335,"thread_start_us":1212,"threads_started":1}
I20260619 13:00:23.847750 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.838635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40679 (local address 127.1.87.65:54923)
0619 13:00:23.839605 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:23.839627 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:23.839862 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:23.840335 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:23.840338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:23.840356 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:23.840828 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:23.840845 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.842341 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.842374 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:23.844963 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:23.844977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.846624 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.846630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:23.846890 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.846948 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 13:00:23.847036 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:00:23.847241 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":296,"thread_start_us":991,"threads_started":1}
I20260619 13:00:23.847772 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.838664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36671 (local address 127.1.87.65:42427)
0619 13:00:23.839897 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:23.839914 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:23.840006 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:23.840499 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:23.840505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:23.840533 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:23.840978 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:23.840990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.842423 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.842428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:23.844963 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:23.844977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.846634 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.846640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:23.846838 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.846941 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 13:00:23.847036 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 13:00:23.847241 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1110,"mutex_wait_us":248,"thread_start_us":648,"threads_started":1}
I20260619 13:00:23.848366 3426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a89b2bdecbf49a58567dee13218e384" candidate_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6796dd76bd9740ae844e6f681546cd0c" is_pre_election: true
I20260619 13:00:23.848418 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a89b2bdecbf49a58567dee13218e384" candidate_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15a5c567d04fbd92a402b42f0a220f" is_pre_election: true
I20260619 13:00:23.848615 3426 raft_consensus.cc:2454] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb54da8bae14050a9d1f5c2d60d9e97 in term 0.
I20260619 13:00:23.848666 3569 raft_consensus.cc:2454] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb54da8bae14050a9d1f5c2d60d9e97 in term 0.
I20260619 13:00:23.849099 3217 leader_election.cc:302] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15a5c567d04fbd92a402b42f0a220f, eeb54da8bae14050a9d1f5c2d60d9e97; no voters:
I20260619 13:00:23.849521 3736 raft_consensus.cc:2781] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:23.849633 3736 raft_consensus.cc:482] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:23.849706 3736 raft_consensus.cc:3037] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:23.905194 3736 raft_consensus.cc:504] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.906149 3736 leader_election.cc:288] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [CANDIDATE]: Term 1 election: Requested vote from peers 6796dd76bd9740ae844e6f681546cd0c (127.1.87.66:40679), 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671)
I20260619 13:00:23.906873 3426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a89b2bdecbf49a58567dee13218e384" candidate_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6796dd76bd9740ae844e6f681546cd0c"
I20260619 13:00:23.906914 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a89b2bdecbf49a58567dee13218e384" candidate_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c15a5c567d04fbd92a402b42f0a220f"
I20260619 13:00:23.907122 3426 raft_consensus.cc:3037] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:23.907166 3569 raft_consensus.cc:3037] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:23.909185 3426 raft_consensus.cc:2454] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb54da8bae14050a9d1f5c2d60d9e97 in term 1.
I20260619 13:00:23.909686 3216 leader_election.cc:302] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6796dd76bd9740ae844e6f681546cd0c, eeb54da8bae14050a9d1f5c2d60d9e97; no voters:
I20260619 13:00:23.910012 3736 raft_consensus.cc:2781] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:23.910658 3736 raft_consensus.cc:686] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 LEADER]: Becoming Leader. State: Replica: eeb54da8bae14050a9d1f5c2d60d9e97, State: Running, Role: LEADER
I20260619 13:00:23.911062 3736 consensus_queue.cc:237] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } }
I20260619 13:00:23.914171 3127 catalog_manager.cc:5495] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65). New cstate: current_term: 1 leader_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb54da8bae14050a9d1f5c2d60d9e97" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 } health_report { overall_health: UNKNOWN } } }
I20260619 13:00:23.944348 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.938542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.1.87.65:40749)
0619 13:00:23.939643 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:23.939647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:23.939665 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:00:23.939738 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:23.939741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:23.939807 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:23.939948 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:23.940296 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.941053 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.942231 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.942623 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.942715 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.943069 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:23.943716 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:23.943823 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:23.943922 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 13:00:23.943985 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":862,"threads_started":1}
I20260619 13:00:23.958352 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.953453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:46425 (local address 127.1.87.67:36671)
0619 13:00:23.953609 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:23.953613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:23.953829 (+ 216us) server_negotiation.cc:407] Connection header received
0619 13:00:23.953960 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:23.953964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:23.954024 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:23.954096 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:23.954565 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.955295 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.956666 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.957088 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.957233 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.957670 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:23.957770 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:23.957865 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:23.957996 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 13:00:23.958064 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 13:00:23.958470 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.953155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36671 (local address 127.1.87.65:46425)
0619 13:00:23.953743 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:23.953762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:23.953867 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:23.954123 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:23.954126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:23.954134 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:00:23.954411 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:23.954419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.955472 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.955479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:23.956528 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:23.956539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.957266 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.957272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:23.957405 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.957452 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:00:23.957866 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:00:23.957879 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 13:00:23.957977 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 13:00:23.958097 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":290,"threads_started":1}
I20260619 13:00:23.966719 3426 raft_consensus.cc:1261] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Refusing update from remote peer eeb54da8bae14050a9d1f5c2d60d9e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:23.967290 3749 consensus_queue.cc:1035] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6796dd76bd9740ae844e6f681546cd0c" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:23.990916 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.984220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:32901 (local address 127.1.87.65:40749)
0619 13:00:23.984433 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:23.984442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:23.985210 (+ 768us) server_negotiation.cc:407] Connection header received
0619 13:00:23.985451 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:23.985456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:23.985542 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:23.985638 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:23.986015 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.987502 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.989228 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.989685 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.989874 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.990247 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:23.990345 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:23.990440 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:23.990551 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 13:00:23.990621 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:00:23.991026 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:23.984209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40749 (local address 127.1.87.67:32901)
0619 13:00:23.985113 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:23.985128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:23.985274 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:23.985656 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:23.985659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:23.985668 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:00:23.985898 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:23.985905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.987690 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.987700 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:23.989081 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:23.989112 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:23.989892 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:23.989898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:23.990045 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:23.990081 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:00:23.990440 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:00:23.990453 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 13:00:23.990534 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 13:00:23.990663 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":687,"threads_started":1}
W20260619 13:00:23.992434 3703 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 13:00:24.009034 3569 raft_consensus.cc:2454] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb54da8bae14050a9d1f5c2d60d9e97 in term 1.
I20260619 13:00:24.009650 3568 raft_consensus.cc:1261] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Refusing update from remote peer eeb54da8bae14050a9d1f5c2d60d9e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:24.010600 3749 consensus_queue.cc:1035] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c15a5c567d04fbd92a402b42f0a220f" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:25.601652 3128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:25.601922 3128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:25.605803 3128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1493604fe2a4aa4b953f35d8127cd11: Sending DeleteTablet for 3 replicas of tablet 1a89b2bdecbf49a58567dee13218e384
I20260619 13:00:25.606765 3545 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a89b2bdecbf49a58567dee13218e384 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:25 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:58588
I20260619 13:00:25.606555 3265 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a89b2bdecbf49a58567dee13218e384 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:25 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:40856
I20260619 13:00:25.607069 3405 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a89b2bdecbf49a58567dee13218e384 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:25 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:55680
I20260619 13:00:25.607544 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3200
I20260619 13:00:25.631913 3784 tablet_replica.cc:330] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97: stopping tablet replica
I20260619 13:00:25.632263 3784 raft_consensus.cc:2229] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:00:25.633014 3784 raft_consensus.cc:2258] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:25.633746 3783 tablet_replica.cc:330] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: stopping tablet replica
I20260619 13:00:25.633937 3783 raft_consensus.cc:2229] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:25.634196 3783 raft_consensus.cc:2258] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:25.634333 3784 ts_tablet_manager.cc:1905] T 1a89b2bdecbf49a58567dee13218e384 P eeb54da8bae14050a9d1f5c2d60d9e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:25.635579 3783 ts_tablet_manager.cc:1905] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:25.639511 3785 tablet_replica.cc:330] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: stopping tablet replica
I20260619 13:00:25.639726 3785 raft_consensus.cc:2229] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:25.639951 3785 raft_consensus.cc:2258] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:25.640795 3785 ts_tablet_manager.cc:1905] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:00:25 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:25.649480 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.609500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.1.87.65:40749)
0619 13:00:25.630716 (+ 21216us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:25.630726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:25.631746 (+ 1020us) server_negotiation.cc:407] Connection header received
0619 13:00:25.631842 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:25.631850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:25.631958 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:25.632207 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:25.635030 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:25.636695 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:25.639098 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:25.639712 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:25.639876 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:25.644142 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:25.644164 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:25.644181 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:25.644211 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:25.646789 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:25.647445 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:25.647460 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:25.647463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:25.647621 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:25.647977 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:25.647981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:25.647985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:25.648320 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:25.648648 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:25.648917 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 13:00:25.649132 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20973,"thread_start_us":21252,"threads_started":1}
W20260619 13:00:25.650158 3331 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42554
W20260619 13:00:25.650808 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3200: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:00:25.651022 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 3200
I20260619 13:00:25.704663 3785 ts_tablet_manager.cc:1918] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:00:25.704665 3783 ts_tablet_manager.cc:1918] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:00:25.704833 3783 log.cc:1196] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/1a89b2bdecbf49a58567dee13218e384
I20260619 13:00:25.704986 3785 log.cc:1196] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/1a89b2bdecbf49a58567dee13218e384
I20260619 13:00:25.705317 3783 ts_tablet_manager.cc:1939] T 1a89b2bdecbf49a58567dee13218e384 P 2c15a5c567d04fbd92a402b42f0a220f: Deleting consensus metadata
I20260619 13:00:25.705551 3785 ts_tablet_manager.cc:1939] T 1a89b2bdecbf49a58567dee13218e384 P 6796dd76bd9740ae844e6f681546cd0c: Deleting consensus metadata
W20260619 13:00:25.727437 3115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.87.65:40749 (error 108)
I20260619 13:00:25.727617 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3340
I20260619 13:00:25.729768 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.728793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40749 (local address 127.0.0.1:42562)
0619 13:00:25.729355 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:25.729522 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":250,"threads_started":1}
W20260619 13:00:25.729991 3115 catalog_manager.cc:4584] TS eeb54da8bae14050a9d1f5c2d60d9e97 (127.1.87.65:40749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a89b2bdecbf49a58567dee13218e384: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40749: connect: Connection refused (error 111)
Jun 19 13:00:25 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:00:25.743134 3789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.728448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.1.87.66:40679)
0619 13:00:25.728965 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:25.728974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:25.728999 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:00:25.729076 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:25.729081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:25.729171 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:25.729379 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:25.730710 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:25.731511 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:25.732763 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:25.733151 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:25.733298 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:25.737361 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:25.737375 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:25.737391 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:25.737423 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:25.740498 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:25.741163 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:25.741170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:25.741172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:25.741291 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:25.741652 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:25.741659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:25.741664 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:25.742010 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:25.742307 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:25.742582 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 13:00:25.742802 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":259,"threads_started":1}
W20260619 13:00:25.743829 3470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40766
W20260619 13:00:25.744503 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3340: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:00:25.744652 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 3340
I20260619 13:00:25.773366 3113 catalog_manager.cc:4841] TS 2c15a5c567d04fbd92a402b42f0a220f (127.1.87.67:36671): tablet 1a89b2bdecbf49a58567dee13218e384 (table test-table [id=cb129f7d3f4940a19d373e3952f2f842]) successfully deleted
I20260619 13:00:25.783586 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.783099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40749 (local address 127.0.0.1:42570)
0619 13:00:25.783287 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:25.783435 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 13:00:25.784552 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.784197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40749 (local address 127.0.0.1:42578)
0619 13:00:25.784345 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:25.784426 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 13:00:25.794322 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3479
I20260619 13:00:25.795310 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.794619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40679 (local address 127.0.0.1:40780)
0619 13:00:25.794775 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:25.795069 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.66:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
Jun 19 13:00:25 dist-test-slave-tpsm krb5kdc[3083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:00:25.811220 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.795123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.1.87.67:36671)
0619 13:00:25.795749 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:25.795753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:25.795769 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:00:25.795819 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:25.795823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:25.795884 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:25.796002 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:25.797229 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:25.797961 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:25.799212 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:25.799701 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:25.799834 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:25.804622 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:25.804637 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:25.804665 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:25.804704 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:25.808083 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:25.809048 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:25.809057 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:25.809059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:25.809231 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:25.809647 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:25.809654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:25.809658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:25.810029 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:25.810338 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:25.810634 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 13:00:25.810864 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":332,"threads_started":1}
W20260619 13:00:25.811923 3609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60072
W20260619 13:00:25.812500 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3479: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:00:25.812670 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 3479
I20260619 13:00:25.828050 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.827589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40679 (local address 127.0.0.1:40786)
0619 13:00:25.827779 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:25.827861 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.66:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 13:00:25.829418 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.829008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40679 (local address 127.0.0.1:40798)
0619 13:00:25.829219 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:25.829310 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.66:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 13:00:25.839895 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3096
I20260619 13:00:25.841125 3168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:25.954330 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:25.953659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40679 (local address 127.0.0.1:40808)
0619 13:00:25.954029 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:25.954118 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.66:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
I20260619 13:00:25.954705 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 3096
2026-06-19T13:00:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8660 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:27 dist-test-slave-tpsm krb5kdc[3796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:27 dist-test-slave-tpsm krb5kdc[3796](info): set up 2 sockets
Jun 19 13:00:27 dist-test-slave-tpsm krb5kdc[3796](info): commencing operation
krb5kdc: starting...
W20260619 13:00:28.951592 1373 mini_kdc.cc:121] Time spent starting KDC: real 2.951s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:29 dist-test-slave-tpsm krb5kdc[3796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:29.133092 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:46605
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:46605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:29.463457 3809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:29.463748 3809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:29.463922 3809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:29.471002 3809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:29.471099 3809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:29.471212 3809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:29.471318 3809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:29.471421 3809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:29.481117 3809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:46605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:46605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:29.482640 3809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:29.484005 3809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:29.495638 3815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:29.495750 3814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:29.495767 3817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:29.496438 3809 server_base.cc:1034] running on GCE node
I20260619 13:00:29.497262 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:29.498757 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:29.499989 3809 hybrid_clock.cc:648] HybridClock initialized: now 1781874029499953 us; error 66 us; skew 500 ppm
I20260619 13:00:29.500375 3809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:29 dist-test-slave-tpsm krb5kdc[3796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874029, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:29.506309 3809 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:29.507781 3809 webserver.cc:460] Webserver started at http://127.1.87.126:46517/ using document root <none> and password file <none>
I20260619 13:00:29.508440 3809 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:29.508581 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:29.508890 3809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:29.511799 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe"
format_stamp: "Formatted at 2026-06-19 13:00:29 on dist-test-slave-tpsm"
server_key: "3093de9fd046a83c90b4a55a6924ea5b"
server_key_iv: "0347c4f3982deefc8202979c548e5174"
server_key_version: "encryptionkey@0"
I20260619 13:00:29.512622 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe"
format_stamp: "Formatted at 2026-06-19 13:00:29 on dist-test-slave-tpsm"
server_key: "3093de9fd046a83c90b4a55a6924ea5b"
server_key_iv: "0347c4f3982deefc8202979c548e5174"
server_key_version: "encryptionkey@0"
I20260619 13:00:29.517541 3809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 13:00:29.520927 3824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:29.522239 3809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:00:29.522419 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe"
format_stamp: "Formatted at 2026-06-19 13:00:29 on dist-test-slave-tpsm"
server_key: "3093de9fd046a83c90b4a55a6924ea5b"
server_key_iv: "0347c4f3982deefc8202979c548e5174"
server_key_version: "encryptionkey@0"
I20260619 13:00:29.522619 3809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:29.539022 3809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:29.542825 3809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:29.543099 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:29.561791 3809 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:46605
I20260619 13:00:29.561812 3875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:46605 every 8 connection(s)
I20260619 13:00:29.563098 3809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:29.566787 3876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:29.569924 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 3809
I20260619 13:00:29.570151 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:29.570907 1373 external_mini_cluster.cc:1428] Setting key 1ab9f4b5fa6c8216ba9e8f70430ec071
I20260619 13:00:29.579325 3876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe: Bootstrap starting.
I20260619 13:00:29.582952 3876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:29.584254 3876 log.cc:824] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:00:29 dist-test-slave-tpsm krb5kdc[3796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:29.587276 3876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe: No bootstrap required, opened a new log
I20260619 13:00:29.594033 3879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:29.572969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.1.87.126:46605)
0619 13:00:29.573678 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:29.573688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:29.573727 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:00:29.575353 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:29.575382 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:29.575755 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:29.576055 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:29.577711 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:29.578904 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:29.580608 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:29.581260 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:29.581380 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:29.586091 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:29.586099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:29.586114 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:29.586149 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:29.589350 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:29.590082 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:29.590089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:29.590095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:29.590260 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:29.590649 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:29.590654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:29.590658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:29.591229 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:29.591593 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:29.592641 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0619 13:00:29.592982 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":339,"threads_started":1}
I20260619 13:00:29.595851 3876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46605 } }
I20260619 13:00:29.596310 3876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:29.596417 3876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 624ca3ca5e8640c2b50a6bb9cda50bbe, State: Initialized, Role: FOLLOWER
I20260619 13:00:29.597025 3876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46605 } }
I20260619 13:00:29.597290 3876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:29.597433 3876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:29.597599 3876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:29.599916 3876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46605 } }
I20260619 13:00:29.600562 3876 leader_election.cc:302] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 624ca3ca5e8640c2b50a6bb9cda50bbe; no voters:
I20260619 13:00:29.601184 3876 leader_election.cc:288] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:29.601336 3881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:29.602620 3881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [term 1 LEADER]: Becoming Leader. State: Replica: 624ca3ca5e8640c2b50a6bb9cda50bbe, State: Running, Role: LEADER
I20260619 13:00:29.603207 3881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46605 } }
I20260619 13:00:29.603578 3876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:29.606245 3883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 624ca3ca5e8640c2b50a6bb9cda50bbe. Latest consensus state: current_term: 1 leader_uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46605 } } }
I20260619 13:00:29.606889 3883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:29.606272 3882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624ca3ca5e8640c2b50a6bb9cda50bbe" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46605 } } }
I20260619 13:00:29.608008 3882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:29.609402 3887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:29.614634 3887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:29.624112 3887 catalog_manager.cc:1347] Generated new cluster ID: a1744d16ef634bc19d1066911f38193f
I20260619 13:00:29.624225 3887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:29.651121 3887 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:29.652474 3887 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:29.662454 3887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 624ca3ca5e8640c2b50a6bb9cda50bbe: Generated new TSK 0
I20260619 13:00:29.663579 3887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:00:29.707690 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46605
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:30.034574 3902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:30.034889 3902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:30.035048 3902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:30.042824 3902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:30.042965 3902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:30.043133 3902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:30.052402 3902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:30.053629 3902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:30.054879 3902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:30.067389 3907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:30.067350 3910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:30.067430 3908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:30.068217 3902 server_base.cc:1034] running on GCE node
I20260619 13:00:30.068820 3902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:30.069692 3902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:30.070921 3902 hybrid_clock.cc:648] HybridClock initialized: now 1781874030070840 us; error 74 us; skew 500 ppm
I20260619 13:00:30.071218 3902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:30 dist-test-slave-tpsm krb5kdc[3796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874030, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:30.074294 3902 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:00:30.075817 3902 webserver.cc:460] Webserver started at http://127.1.87.65:39921/ using document root <none> and password file <none>
I20260619 13:00:30.076476 3902 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:30.076618 3902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:30.076924 3902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:30.079758 3902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "b6bbc67bd91e408cb141b23d039b74d7"
format_stamp: "Formatted at 2026-06-19 13:00:30 on dist-test-slave-tpsm"
server_key: "126ab982f8972b9b1a6282ce4599bceb"
server_key_iv: "df427459134d36b29bd7e45240519c3d"
server_key_version: "encryptionkey@0"
I20260619 13:00:30.080555 3902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "b6bbc67bd91e408cb141b23d039b74d7"
format_stamp: "Formatted at 2026-06-19 13:00:30 on dist-test-slave-tpsm"
server_key: "126ab982f8972b9b1a6282ce4599bceb"
server_key_iv: "df427459134d36b29bd7e45240519c3d"
server_key_version: "encryptionkey@0"
I20260619 13:00:30.085564 3902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:00:30.088901 3917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:30.090106 3902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:00:30.090272 3902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "b6bbc67bd91e408cb141b23d039b74d7"
format_stamp: "Formatted at 2026-06-19 13:00:30 on dist-test-slave-tpsm"
server_key: "126ab982f8972b9b1a6282ce4599bceb"
server_key_iv: "df427459134d36b29bd7e45240519c3d"
server_key_version: "encryptionkey@0"
I20260619 13:00:30.090472 3902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:30.107257 3902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:30.111604 3902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:30.111901 3902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:30.112802 3902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:30.114048 3902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:30.114154 3902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:30.114305 3902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:30.114377 3902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:30.147181 3902 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:40057
I20260619 13:00:30.147235 4029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:40057 every 8 connection(s)
I20260619 13:00:30.148517 3902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:30.157078 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 3902
I20260619 13:00:30.157302 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:30.157902 1373 external_mini_cluster.cc:1428] Setting key 384093a8d2bd01b13048a8e46fb396c1
Jun 19 13:00:30 dist-test-slave-tpsm krb5kdc[3796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874030, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:30.168632 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:30.151055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:55819 (local address 127.1.87.126:46605)
0619 13:00:30.151542 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:30.151546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:30.152476 (+ 930us) server_negotiation.cc:407] Connection header received
0619 13:00:30.154236 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:30.154242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:30.154328 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:30.154444 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:30.155909 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:30.156687 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:30.158342 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:30.158788 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:30.158929 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:30.162730 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:30.162738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:30.162745 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:30.162779 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:30.165219 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:30.166099 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:30.166107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:30.166113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:30.166257 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:30.166646 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:30.166654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:30.166660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:30.167026 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:30.167179 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:30.167903 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 13:00:30.168152 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":281,"threads_started":1}
I20260619 13:00:30.169166 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:30.151613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:46605 (local address 127.1.87.65:55819)
0619 13:00:30.152306 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:30.152347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:30.153963 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:30.154667 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:30.154675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:30.155155 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:30.155749 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:30.155765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:30.156835 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:30.156838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:30.158210 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:30.158221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:30.159000 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:30.159005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:30.159186 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:30.159883 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:30.159912 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:30.162473 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:30.165346 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:30.165356 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:30.165375 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:30.165978 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:30.166357 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:30.166360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:30.166363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:30.166539 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:30.167141 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:30.167147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:30.167528 (+ 381us) client_negotiation.cc:769] Sending connection context
0619 13:00:30.167774 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 13:00:30.168123 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":266,"threads_started":1}
I20260619 13:00:30.170629 4030 heartbeater.cc:347] Connected to a master server at 127.1.87.126:46605
I20260619 13:00:30.170935 4030 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:30.171809 4030 heartbeater.cc:511] Master 127.1.87.126:46605 requested a full tablet report, sending...
I20260619 13:00:30.174072 3841 ts_manager.cc:194] Registered new tserver with Master: b6bbc67bd91e408cb141b23d039b74d7 (127.1.87.65:40057)
I20260619 13:00:30.175552 3841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:55819
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:00:30.198267 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46605
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:30.514788 4036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:30.515056 4036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:30.515148 4036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:30.522264 4036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:30.522379 4036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:30.522492 4036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:30.532378 4036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:30.533571 4036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:30.534860 4036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:30.548030 4044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:30.548122 4041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:30.548095 4042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:30.548979 4036 server_base.cc:1034] running on GCE node
I20260619 13:00:30.549575 4036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:30.550607 4036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:30.551841 4036 hybrid_clock.cc:648] HybridClock initialized: now 1781874030551799 us; error 29 us; skew 500 ppm
I20260619 13:00:30.552165 4036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:30 dist-test-slave-tpsm krb5kdc[3796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874030, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:30.555238 4036 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:00:30.556605 4036 webserver.cc:460] Webserver started at http://127.1.87.66:44097/ using document root <none> and password file <none>
I20260619 13:00:30.557278 4036 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:30.557420 4036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:30.557734 4036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:30.560549 4036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "97776646068a44d9a59abe7e153084fc"
format_stamp: "Formatted at 2026-06-19 13:00:30 on dist-test-slave-tpsm"
server_key: "0fe09908190a8acdd5f3e2b98dfeaf8e"
server_key_iv: "d51a82cbf38e9092226abc95c53a89ab"
server_key_version: "encryptionkey@0"
I20260619 13:00:30.561542 4036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "97776646068a44d9a59abe7e153084fc"
format_stamp: "Formatted at 2026-06-19 13:00:30 on dist-test-slave-tpsm"
server_key: "0fe09908190a8acdd5f3e2b98dfeaf8e"
server_key_iv: "d51a82cbf38e9092226abc95c53a89ab"
server_key_version: "encryptionkey@0"
I20260619 13:00:30.567886 4036 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 13:00:30.571945 4051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:30.573390 4036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:00:30.573537 4036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "97776646068a44d9a59abe7e153084fc"
format_stamp: "Formatted at 2026-06-19 13:00:30 on dist-test-slave-tpsm"
server_key: "0fe09908190a8acdd5f3e2b98dfeaf8e"
server_key_iv: "d51a82cbf38e9092226abc95c53a89ab"
server_key_version: "encryptionkey@0"
I20260619 13:00:30.573766 4036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:30.592511 4036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:30.596853 4036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:30.597097 4036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:30.597967 4036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:30.599251 4036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:30.599480 4036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:30.599588 4036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:30.599634 4036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:30.632069 4036 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:42321
I20260619 13:00:30.632121 4163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:42321 every 8 connection(s)
I20260619 13:00:30.633252 4036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:30.635607 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 4036
I20260619 13:00:30.635804 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:30.636238 1373 external_mini_cluster.cc:1428] Setting key 25cab3223320a0e7ffd9c893a7d485a4
Jun 19 13:00:30 dist-test-slave-tpsm krb5kdc[3796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874030, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:30.653596 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:30.635709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:58107 (local address 127.1.87.126:46605)
0619 13:00:30.635909 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:30.635915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:30.637164 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 13:00:30.638953 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:30.638961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:30.639049 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:30.639172 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:30.640535 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:30.641698 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:30.643154 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:30.643906 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:30.644052 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:30.647860 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:30.647865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:30.647868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:30.647899 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:30.650405 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:30.651401 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:30.651405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:30.651407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:30.651533 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:30.652028 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:30.652032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:30.652034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:30.652273 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:30.652376 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:30.653208 (+ 832us) server_negotiation.cc:299] Negotiation successful
0619 13:00:30.653357 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:00:30.654894 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:30.636192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:46605 (local address 127.1.87.66:58107)
0619 13:00:30.637007 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:30.637062 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:30.638702 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:30.639390 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:30.639398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:30.639874 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:30.640373 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:30.640388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:30.641846 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:30.641850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:30.643009 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:30.643020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:30.644132 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:30.644138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:30.644327 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:30.644990 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:30.645015 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:30.647619 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:30.650542 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:30.650554 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:30.650572 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:30.651236 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:30.651660 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:30.651668 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:30.651673 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:30.651907 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:30.652401 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:30.652411 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:30.652830 (+ 419us) client_negotiation.cc:769] Sending connection context
0619 13:00:30.653142 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 13:00:30.653681 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
I20260619 13:00:30.656389 4164 heartbeater.cc:347] Connected to a master server at 127.1.87.126:46605
I20260619 13:00:30.656719 4164 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:30.657577 4164 heartbeater.cc:511] Master 127.1.87.126:46605 requested a full tablet report, sending...
I20260619 13:00:30.659142 3841 ts_manager.cc:194] Registered new tserver with Master: 97776646068a44d9a59abe7e153084fc (127.1.87.66:42321)
I20260619 13:00:30.659978 3841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:58107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:00:30.677693 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46605
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:30.994385 4169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:30.994699 4169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:30.994889 4169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:31.002027 4169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:31.002185 4169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:31.002375 4169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:31.012231 4169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:31.013471 4169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:31.014694 4169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:31.027422 4175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:31.027442 4174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:31.028249 4177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:31.033980 4169 server_base.cc:1034] running on GCE node
I20260619 13:00:31.034600 4169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:31.035585 4169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:31.036795 4169 hybrid_clock.cc:648] HybridClock initialized: now 1781874031036743 us; error 37 us; skew 500 ppm
I20260619 13:00:31.037117 4169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:31 dist-test-slave-tpsm krb5kdc[3796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:31.040231 4169 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:00:31.041685 4169 webserver.cc:460] Webserver started at http://127.1.87.67:32991/ using document root <none> and password file <none>
I20260619 13:00:31.042322 4169 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:31.042416 4169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:31.042662 4169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:31.045464 4169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "21d9e0ce1d1e4dba9698b6246557bb35"
format_stamp: "Formatted at 2026-06-19 13:00:31 on dist-test-slave-tpsm"
server_key: "aa857fae073b09f46d270bf013ac6751"
server_key_iv: "5551981cd9c449e42b007aec61838c50"
server_key_version: "encryptionkey@0"
I20260619 13:00:31.046130 4169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "21d9e0ce1d1e4dba9698b6246557bb35"
format_stamp: "Formatted at 2026-06-19 13:00:31 on dist-test-slave-tpsm"
server_key: "aa857fae073b09f46d270bf013ac6751"
server_key_iv: "5551981cd9c449e42b007aec61838c50"
server_key_version: "encryptionkey@0"
I20260619 13:00:31.050865 4169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 13:00:31.054214 4184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:31.055577 4169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:00:31.055729 4169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "21d9e0ce1d1e4dba9698b6246557bb35"
format_stamp: "Formatted at 2026-06-19 13:00:31 on dist-test-slave-tpsm"
server_key: "aa857fae073b09f46d270bf013ac6751"
server_key_iv: "5551981cd9c449e42b007aec61838c50"
server_key_version: "encryptionkey@0"
I20260619 13:00:31.055922 4169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:31.089080 4169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:31.093281 4169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:31.093559 4169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:31.094545 4169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:31.095873 4169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:31.095942 4169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:31.096098 4169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:31.096150 4169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:31.138329 4169 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:39257
I20260619 13:00:31.138348 4296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:39257 every 8 connection(s)
I20260619 13:00:31.139659 4169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:31.146675 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 4169
I20260619 13:00:31.146878 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:31.147495 1373 external_mini_cluster.cc:1428] Setting key 80af55842d1123de470d21da39864d7b
Jun 19 13:00:31 dist-test-slave-tpsm krb5kdc[3796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:31.164042 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:31.142215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:42191 (local address 127.1.87.126:46605)
0619 13:00:31.142417 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:31.142423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:31.143735 (+ 1312us) server_negotiation.cc:407] Connection header received
0619 13:00:31.145521 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:31.145530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:31.145636 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:31.145775 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:31.147670 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:31.148829 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:31.151202 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:31.152114 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:31.152264 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:31.155916 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:31.155925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:31.155933 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:31.155982 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:31.160031 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:31.160810 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:31.160818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:31.160825 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:31.160983 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:31.161542 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:31.161549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:31.161555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:31.161962 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:31.162123 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:31.163507 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0619 13:00:31.163734 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:00:31.164456 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:31.142748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:46605 (local address 127.1.87.67:42191)
0619 13:00:31.143563 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:31.143605 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:31.145230 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:31.145938 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:31.145945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:31.146725 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:31.147459 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:31.147482 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:31.149007 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:31.149012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:31.151041 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:31.151055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:31.152354 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:31.152361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:31.152551 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:31.153217 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:31.153239 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:31.155684 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:31.160168 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:31.160178 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:31.160199 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:31.160657 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:31.161136 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:31.161144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:31.161149 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:31.161409 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:31.162096 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:31.162103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:31.162473 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 13:00:31.162779 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 13:00:31.163182 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":357,"threads_started":1}
I20260619 13:00:31.165719 4297 heartbeater.cc:347] Connected to a master server at 127.1.87.126:46605
I20260619 13:00:31.166071 4297 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:31.166980 4297 heartbeater.cc:511] Master 127.1.87.126:46605 requested a full tablet report, sending...
I20260619 13:00:31.168572 3841 ts_manager.cc:194] Registered new tserver with Master: 21d9e0ce1d1e4dba9698b6246557bb35 (127.1.87.67:39257)
I20260619 13:00:31.169410 3841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:42191
I20260619 13:00:31.175436 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:00:31.178431 4030 heartbeater.cc:503] Master 127.1.87.126:46605 was elected leader, sending a full tablet report...
W20260619 13:00:31.211651 4032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:00:31.208946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.1.87.126:46605)
0619 13:00:31.209194 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:31.209204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:31.209611 (+ 407us) server_negotiation.cc:407] Connection header received
0619 13:00:31.209763 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:31.209772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:31.209876 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:31.210038 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:31.211296 (+ 1258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39848: BlockingRecv error: recv got EOF from 127.0.0.1:39848 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:00:31.214643 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3902
W20260619 13:00:31.218036 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3902: Not authorized: Client connection negotiation failed: client connection to 127.1.87.65:40057: server requires authentication, but client does not have Kerberos credentials available
I20260619 13:00:31.218218 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 3902
I20260619 13:00:31.241752 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4036
W20260619 13:00:31.244653 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4036: Not authorized: Client connection negotiation failed: client connection to 127.1.87.66:42321: server requires authentication, but client does not have Kerberos credentials available
I20260619 13:00:31.244807 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 4036
I20260619 13:00:31.260216 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4169
W20260619 13:00:31.262974 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4169: Not authorized: Client connection negotiation failed: client connection to 127.1.87.67:39257: server requires authentication, but client does not have Kerberos credentials available
I20260619 13:00:31.263110 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 4169
I20260619 13:00:31.284900 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 3809
I20260619 13:00:31.286012 3871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:31.385669 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 3809
2026-06-19T13:00:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5426 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:31 dist-test-slave-tpsm krb5kdc[4314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:31 dist-test-slave-tpsm krb5kdc[4314](info): set up 2 sockets
Jun 19 13:00:31 dist-test-slave-tpsm krb5kdc[4314](info): commencing operation
krb5kdc: starting...
W20260619 13:00:32.483616 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:32 dist-test-slave-tpsm krb5kdc[4314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:32.676697 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44981
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:44981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:32.988675 4327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:32.989044 4327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:32.989217 4327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:32.996665 4327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:32.996853 4327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:32.996966 4327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:32.997074 4327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:32.997166 4327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:33.007498 4327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:44981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:33.008848 4327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:33.010149 4327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:33.022449 4332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:33.022379 4335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:33.022418 4333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:33.023701 4327 server_base.cc:1034] running on GCE node
I20260619 13:00:33.024539 4327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:33.026055 4327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:33.027398 4327 hybrid_clock.cc:648] HybridClock initialized: now 1781874033027305 us; error 118 us; skew 500 ppm
I20260619 13:00:33.027786 4327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:33 dist-test-slave-tpsm krb5kdc[4314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874033, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:33.030790 4327 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:33.032227 4327 webserver.cc:460] Webserver started at http://127.1.87.126:39533/ using document root <none> and password file <none>
I20260619 13:00:33.032840 4327 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:33.032938 4327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:33.033195 4327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:33.036160 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "033150d954b643dbaf939cf9bd4a9f10"
format_stamp: "Formatted at 2026-06-19 13:00:33 on dist-test-slave-tpsm"
server_key: "48e187c28e1512fdd47c5a7a81e2a350"
server_key_iv: "dc06a8e62e0177dd3aeb015e1d164a40"
server_key_version: "encryptionkey@0"
I20260619 13:00:33.036943 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "033150d954b643dbaf939cf9bd4a9f10"
format_stamp: "Formatted at 2026-06-19 13:00:33 on dist-test-slave-tpsm"
server_key: "48e187c28e1512fdd47c5a7a81e2a350"
server_key_iv: "dc06a8e62e0177dd3aeb015e1d164a40"
server_key_version: "encryptionkey@0"
I20260619 13:00:33.041828 4327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:00:33.045315 4342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:33.046600 4327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:33.046741 4327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "033150d954b643dbaf939cf9bd4a9f10"
format_stamp: "Formatted at 2026-06-19 13:00:33 on dist-test-slave-tpsm"
server_key: "48e187c28e1512fdd47c5a7a81e2a350"
server_key_iv: "dc06a8e62e0177dd3aeb015e1d164a40"
server_key_version: "encryptionkey@0"
I20260619 13:00:33.046909 4327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:33.059216 4327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:33.064033 4327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:33.064282 4327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:33.083551 4327 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:44981
I20260619 13:00:33.083573 4393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:44981 every 8 connection(s)
I20260619 13:00:33.084957 4327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:33.088703 4394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:33.094480 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 4327
I20260619 13:00:33.094672 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:33.095377 1373 external_mini_cluster.cc:1428] Setting key 62cbade8a43f38d7fe567050abc8897a
I20260619 13:00:33.099704 4394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10: Bootstrap starting.
I20260619 13:00:33.103431 4394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:33.104646 4394 log.cc:824] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:33.107949 4394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10: No bootstrap required, opened a new log
Jun 19 13:00:33 dist-test-slave-tpsm krb5kdc[4314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:33.115835 4394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033150d954b643dbaf939cf9bd4a9f10" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44981 } }
I20260619 13:00:33.116226 4394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:33.116361 4394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033150d954b643dbaf939cf9bd4a9f10, State: Initialized, Role: FOLLOWER
I20260619 13:00:33.117062 4394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033150d954b643dbaf939cf9bd4a9f10" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44981 } }
I20260619 13:00:33.117334 4394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:33.117465 4394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:33.117647 4394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:33.118608 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:33.097255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.1.87.126:44981)
0619 13:00:33.098073 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:33.098084 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:33.098121 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:00:33.099854 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:33.099887 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:33.100252 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:33.100546 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:33.102381 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:33.103822 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:33.105586 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:33.106364 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:33.106505 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:33.110967 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:33.110975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:33.110989 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:33.111029 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:33.114129 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:33.114803 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:33.114809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:33.114815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:33.114973 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:33.115365 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:33.115372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:33.115374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:33.116044 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:33.116375 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:33.116822 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 13:00:33.117244 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":392,"threads_started":1}
I20260619 13:00:33.120637 4394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033150d954b643dbaf939cf9bd4a9f10" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44981 } }
I20260619 13:00:33.121145 4394 leader_election.cc:302] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 033150d954b643dbaf939cf9bd4a9f10; no voters:
I20260619 13:00:33.121825 4394 leader_election.cc:288] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:33.121956 4399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:33.122611 4399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [term 1 LEADER]: Becoming Leader. State: Replica: 033150d954b643dbaf939cf9bd4a9f10, State: Running, Role: LEADER
I20260619 13:00:33.122788 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:00:33.123179 4399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033150d954b643dbaf939cf9bd4a9f10" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44981 } }
I20260619 13:00:33.125792 4394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:33.126529 4401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "033150d954b643dbaf939cf9bd4a9f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033150d954b643dbaf939cf9bd4a9f10" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44981 } } }
I20260619 13:00:33.126925 4401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:33.128189 4400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 033150d954b643dbaf939cf9bd4a9f10. Latest consensus state: current_term: 1 leader_uuid: "033150d954b643dbaf939cf9bd4a9f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033150d954b643dbaf939cf9bd4a9f10" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44981 } } }
I20260619 13:00:33.128713 4400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:33.131143 4403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:33.137329 4403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:33.162802 4403 catalog_manager.cc:1347] Generated new cluster ID: 6f4acecb800c49b9b88da10228a8f4b9
I20260619 13:00:33.162922 4403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:33.172590 4403 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:33.174024 4403 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:33.179934 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:33.169001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.1.87.126:44981)
0619 13:00:33.169188 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:33.169194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:33.171503 (+ 2309us) server_negotiation.cc:407] Connection header received
0619 13:00:33.171745 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:33.171750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:33.171819 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:33.171924 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:33.173158 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:33.174326 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:33.176175 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:33.176888 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:33.177032 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:33.178193 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:33.178200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:33.178205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:33.179016 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:33.179089 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:33.179122 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:33.179240 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:33.179387 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 13:00:33.179518 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":67}
I20260619 13:00:33.184041 4403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 033150d954b643dbaf939cf9bd4a9f10: Generated new TSK 0
I20260619 13:00:33.185040 4403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:00:33.195832 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4327
I20260619 13:00:33.197078 4389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:33.287088 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 4327
2026-06-19T13:00:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1896 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:33 dist-test-slave-tpsm krb5kdc[4428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:33 dist-test-slave-tpsm krb5kdc[4428](info): set up 2 sockets
Jun 19 13:00:33 dist-test-slave-tpsm krb5kdc[4428](info): commencing operation
krb5kdc: starting...
W20260619 13:00:34.363775 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:34 dist-test-slave-tpsm krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:34.557118 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:46479
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:34.870666 4441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:34.870936 4441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:34.871022 4441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:34.878271 4441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:34.878378 4441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:34.878424 4441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:34.878484 4441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:34.878528 4441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:34.888219 4441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:46479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:34.889374 4441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:34.890523 4441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:34.901636 4447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:34.901603 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:34.901678 4446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:34.904637 4441 server_base.cc:1034] running on GCE node
I20260619 13:00:34.905264 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:34.906452 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:34.907652 4441 hybrid_clock.cc:648] HybridClock initialized: now 1781874034907571 us; error 60 us; skew 500 ppm
I20260619 13:00:34.908077 4441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:34 dist-test-slave-tpsm krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:34.915649 4441 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:34.917148 4441 webserver.cc:460] Webserver started at http://127.1.87.126:35155/ using document root <none> and password file <none>
I20260619 13:00:34.917786 4441 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:34.917886 4441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:34.918140 4441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:34.921003 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "dcc17622d13c496fba6fb7baabb8d557"
format_stamp: "Formatted at 2026-06-19 13:00:34 on dist-test-slave-tpsm"
server_key: "e24c65e429c1bc7928ee208d5843b0df"
server_key_iv: "59a8eda6974c87b5d2932fb076aa7e91"
server_key_version: "encryptionkey@0"
I20260619 13:00:34.921679 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "dcc17622d13c496fba6fb7baabb8d557"
format_stamp: "Formatted at 2026-06-19 13:00:34 on dist-test-slave-tpsm"
server_key: "e24c65e429c1bc7928ee208d5843b0df"
server_key_iv: "59a8eda6974c87b5d2932fb076aa7e91"
server_key_version: "encryptionkey@0"
I20260619 13:00:34.926645 4441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 13:00:34.929909 4456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:34.931155 4441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:34.931321 4441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "dcc17622d13c496fba6fb7baabb8d557"
format_stamp: "Formatted at 2026-06-19 13:00:34 on dist-test-slave-tpsm"
server_key: "e24c65e429c1bc7928ee208d5843b0df"
server_key_iv: "59a8eda6974c87b5d2932fb076aa7e91"
server_key_version: "encryptionkey@0"
I20260619 13:00:34.931546 4441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:34.944680 4441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:34.948643 4441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:34.948927 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:34.966930 4441 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:46479
I20260619 13:00:34.966949 4507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:46479 every 8 connection(s)
I20260619 13:00:34.968348 4441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:34.972123 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:34.975260 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 4441
I20260619 13:00:34.975488 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:34.976264 1373 external_mini_cluster.cc:1428] Setting key c8664fce03eb965302c40aa772699af5
I20260619 13:00:34.985057 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557: Bootstrap starting.
I20260619 13:00:34.988890 4508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:34.990193 4508 log.cc:824] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:00:34 dist-test-slave-tpsm krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:34.993206 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557: No bootstrap required, opened a new log
I20260619 13:00:34.999416 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:34.978278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.1.87.126:46479)
0619 13:00:34.979134 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:34.979154 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:34.979193 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:00:34.980984 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:34.981005 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:34.981370 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:34.981702 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:34.983298 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:34.984801 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:34.986491 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:34.987576 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:34.987723 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:34.992444 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:34.992451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:34.992465 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:34.992506 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:34.995542 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:34.996173 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:34.996191 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:34.996202 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:34.996363 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:34.996750 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:34.996755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:34.996760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:34.997267 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:34.997604 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:34.997972 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 13:00:34.998365 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":457,"threads_started":1}
I20260619 13:00:35.001061 4508 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc17622d13c496fba6fb7baabb8d557" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46479 } }
I20260619 13:00:35.001407 4508 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:35.001528 4508 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc17622d13c496fba6fb7baabb8d557, State: Initialized, Role: FOLLOWER
I20260619 13:00:35.002094 4508 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc17622d13c496fba6fb7baabb8d557" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46479 } }
I20260619 13:00:35.002347 4508 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:35.002477 4508 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:35.002588 4508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:35.004698 4508 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc17622d13c496fba6fb7baabb8d557" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46479 } }
I20260619 13:00:35.005209 4508 leader_election.cc:302] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcc17622d13c496fba6fb7baabb8d557; no voters:
I20260619 13:00:35.005810 4508 leader_election.cc:288] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:35.005939 4513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:35.006939 4513 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [term 1 LEADER]: Becoming Leader. State: Replica: dcc17622d13c496fba6fb7baabb8d557, State: Running, Role: LEADER
I20260619 13:00:35.007558 4513 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc17622d13c496fba6fb7baabb8d557" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46479 } }
I20260619 13:00:35.008033 4508 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:35.010051 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcc17622d13c496fba6fb7baabb8d557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc17622d13c496fba6fb7baabb8d557" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46479 } } }
I20260619 13:00:35.010569 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:35.012166 4518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:35.010248 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcc17622d13c496fba6fb7baabb8d557. Latest consensus state: current_term: 1 leader_uuid: "dcc17622d13c496fba6fb7baabb8d557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc17622d13c496fba6fb7baabb8d557" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46479 } } }
I20260619 13:00:35.012681 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:35.017764 4518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:35.026213 4518 catalog_manager.cc:1347] Generated new cluster ID: fa6fb303d73f408d812c5ab50cb2a165
I20260619 13:00:35.026324 4518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:35.044157 4518 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:35.045624 4518 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:35.057955 4518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcc17622d13c496fba6fb7baabb8d557: Generated new TSK 0
I20260619 13:00:35.058990 4518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:00:35.112950 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46479
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:35.433586 4534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:35.433898 4534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:35.434056 4534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:35.441473 4534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:35.441623 4534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:35.441826 4534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:35.451695 4534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:35.453030 4534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:35.454290 4534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:35.466902 4539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:35.467001 4540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:35.467041 4542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:35.467887 4534 server_base.cc:1034] running on GCE node
I20260619 13:00:35.468448 4534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:35.469312 4534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:35.470521 4534 hybrid_clock.cc:648] HybridClock initialized: now 1781874035470484 us; error 26 us; skew 500 ppm
I20260619 13:00:35.470898 4534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:35 dist-test-slave-tpsm krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874035, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:35.474327 4534 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:00:35.475863 4534 webserver.cc:460] Webserver started at http://127.1.87.65:46631/ using document root <none> and password file <none>
I20260619 13:00:35.476547 4534 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:35.476689 4534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:35.476987 4534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:35.479848 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "1cd0d11a7767413496fcd176bc308a81"
format_stamp: "Formatted at 2026-06-19 13:00:35 on dist-test-slave-tpsm"
server_key: "cfdf7d90a2e39b3a04a0070c908fe127"
server_key_iv: "92fd412e1c011560806d9699c8074eb9"
server_key_version: "encryptionkey@0"
I20260619 13:00:35.480722 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "1cd0d11a7767413496fcd176bc308a81"
format_stamp: "Formatted at 2026-06-19 13:00:35 on dist-test-slave-tpsm"
server_key: "cfdf7d90a2e39b3a04a0070c908fe127"
server_key_iv: "92fd412e1c011560806d9699c8074eb9"
server_key_version: "encryptionkey@0"
I20260619 13:00:35.485960 4534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:00:35.489557 4549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:35.490952 4534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:35.491128 4534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "1cd0d11a7767413496fcd176bc308a81"
format_stamp: "Formatted at 2026-06-19 13:00:35 on dist-test-slave-tpsm"
server_key: "cfdf7d90a2e39b3a04a0070c908fe127"
server_key_iv: "92fd412e1c011560806d9699c8074eb9"
server_key_version: "encryptionkey@0"
I20260619 13:00:35.491407 4534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:35.516959 4534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:35.521628 4534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:35.521937 4534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:35.522886 4534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:35.524310 4534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:35.524415 4534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:35.524556 4534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:35.524632 4534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:35.559582 4534 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:37487
I20260619 13:00:35.559671 4661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:37487 every 8 connection(s)
I20260619 13:00:35.560953 4534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:35.562107 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 4534
I20260619 13:00:35.562320 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:35.563028 1373 external_mini_cluster.cc:1428] Setting key e5f557ba88c9b1102e8a2d26baa5cb0d
Jun 19 13:00:35 dist-test-slave-tpsm krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874035, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
I20260619 13:00:35.581020 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:35.563857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:39329 (local address 127.1.87.126:46479)
0619 13:00:35.564271 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:35.564277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:35.565493 (+ 1216us) server_negotiation.cc:407] Connection header received
0619 13:00:35.567258 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:35.567263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:35.567379 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:35.567496 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:35.568974 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:35.569691 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:35.571210 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:35.571670 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:35.571812 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:35.575405 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:35.575410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:35.575414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:35.575446 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:35.577963 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:35.578699 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:35.578705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:35.578710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:35.578830 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:35.579245 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:35.579252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:35.579255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:35.579576 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:35.579733 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:35.580428 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 13:00:35.580658 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":186,"threads_started":1}
I20260619 13:00:35.582127 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:35.564518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:46479 (local address 127.1.87.65:39329)
0619 13:00:35.565350 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:35.565392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:35.566997 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:35.567749 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:35.567757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:35.568247 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:35.568808 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:35.568825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:35.569842 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:35.569847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:35.571045 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:35.571056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:35.571902 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:35.571911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:35.572142 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:35.572820 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:35.572842 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:35.575155 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:35.578109 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:35.578119 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:35.578138 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:35.578561 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:35.578949 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:35.578954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:35.578958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:35.579139 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:35.579723 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:35.579731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:35.580063 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 13:00:35.580326 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 13:00:35.580884 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":349,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
I20260619 13:00:35.583894 4662 heartbeater.cc:347] Connected to a master server at 127.1.87.126:46479
I20260619 13:00:35.584213 4662 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:35.585029 4662 heartbeater.cc:511] Master 127.1.87.126:46479 requested a full tablet report, sending...
I20260619 13:00:35.588013 4473 ts_manager.cc:194] Registered new tserver with Master: 1cd0d11a7767413496fcd176bc308a81 (127.1.87.65:37487)
I20260619 13:00:35.589740 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:39329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:00:35.606407 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46479
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:35.929854 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:35.930107 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:35.930258 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:35.937484 4668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:35.937589 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:35.937772 4668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:35.947379 4668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:35.948616 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:35.949869 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:35.962431 4676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:35.962482 4674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:35.962503 4673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:35.963306 4668 server_base.cc:1034] running on GCE node
I20260619 13:00:35.963958 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:35.964818 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:35.966074 4668 hybrid_clock.cc:648] HybridClock initialized: now 1781874035966025 us; error 31 us; skew 500 ppm
I20260619 13:00:35.966393 4668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:35 dist-test-slave-tpsm krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874035, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:35.969516 4668 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:00:35.970938 4668 webserver.cc:460] Webserver started at http://127.1.87.66:37965/ using document root <none> and password file <none>
I20260619 13:00:35.971660 4668 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:35.971782 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:35.972085 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:35.974835 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "68919ee147574dd290dc5d8a9000cf75"
format_stamp: "Formatted at 2026-06-19 13:00:35 on dist-test-slave-tpsm"
server_key: "2496691cb37d4d75cbb85c2013f5f3d7"
server_key_iv: "0545ed10795a89ce5450867e56d0b0ae"
server_key_version: "encryptionkey@0"
I20260619 13:00:35.975632 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "68919ee147574dd290dc5d8a9000cf75"
format_stamp: "Formatted at 2026-06-19 13:00:35 on dist-test-slave-tpsm"
server_key: "2496691cb37d4d75cbb85c2013f5f3d7"
server_key_iv: "0545ed10795a89ce5450867e56d0b0ae"
server_key_version: "encryptionkey@0"
I20260619 13:00:35.980479 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 13:00:35.983652 4683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:35.984860 4668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:00:35.985009 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "68919ee147574dd290dc5d8a9000cf75"
format_stamp: "Formatted at 2026-06-19 13:00:35 on dist-test-slave-tpsm"
server_key: "2496691cb37d4d75cbb85c2013f5f3d7"
server_key_iv: "0545ed10795a89ce5450867e56d0b0ae"
server_key_version: "encryptionkey@0"
I20260619 13:00:35.985199 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:36.004652 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:36.008497 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:36.008787 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:36.009775 4668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:36.011029 4668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:36.011107 4668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:36.011256 4668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:36.011348 4668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:36.044077 4668 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:33389
I20260619 13:00:36.044126 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:33389 every 8 connection(s)
I20260619 13:00:36.045264 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:36.055205 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 4668
I20260619 13:00:36.055437 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:36.055924 1373 external_mini_cluster.cc:1428] Setting key 0ebc43369957675fe192760a39dfd9fd
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874035, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:36.065263 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:36.047613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:45727 (local address 127.1.87.126:46479)
0619 13:00:36.047806 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:36.047810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:36.048959 (+ 1149us) server_negotiation.cc:407] Connection header received
0619 13:00:36.050676 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:36.050682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:36.050768 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:36.050875 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:36.052268 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.052995 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.054493 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.055020 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.055152 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:36.058829 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:36.058838 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:36.058846 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:36.058890 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:36.062436 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.063088 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.063096 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.063102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.063250 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.063692 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.063699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.063703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.064082 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:36.064244 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:36.064880 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 13:00:36.065038 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 13:00:36.066704 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:36.048077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:46479 (local address 127.1.87.66:45727)
0619 13:00:36.048797 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:36.048833 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:36.050424 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:36.051096 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:36.051104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:36.051590 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:36.052104 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:36.052120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.053142 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.053150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:36.054313 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:36.054324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.055240 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.055245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:36.055603 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:36.056257 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:36.056282 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:36.058600 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:36.062556 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:36.062563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:36.062575 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:36.062961 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:36.063397 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:36.063401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:36.063403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:36.063568 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:36.064198 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:36.064203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:36.064479 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 13:00:36.064786 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 13:00:36.065332 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":299,"threads_started":1}
I20260619 13:00:36.068120 4796 heartbeater.cc:347] Connected to a master server at 127.1.87.126:46479
I20260619 13:00:36.068435 4796 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:36.069141 4796 heartbeater.cc:511] Master 127.1.87.126:46479 requested a full tablet report, sending...
I20260619 13:00:36.070528 4473 ts_manager.cc:194] Registered new tserver with Master: 68919ee147574dd290dc5d8a9000cf75 (127.1.87.66:33389)
I20260619 13:00:36.071465 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:45727
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:00:36.096545 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46479
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:36.415047 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:36.415405 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:36.415571 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:36.422892 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:36.423034 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:36.423230 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:36.433239 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:46479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:36.434645 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:36.436131 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:36.448987 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:36.448928 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:36.449093 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:36.449987 4801 server_base.cc:1034] running on GCE node
I20260619 13:00:36.450524 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:36.451471 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:36.452713 4801 hybrid_clock.cc:648] HybridClock initialized: now 1781874036452675 us; error 50 us; skew 500 ppm
I20260619 13:00:36.453085 4801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874036, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:36.456169 4801 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:00:36.457655 4801 webserver.cc:460] Webserver started at http://127.1.87.67:34021/ using document root <none> and password file <none>
I20260619 13:00:36.458346 4801 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:36.458492 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:36.458797 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:36.461923 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "aa9070ed926c4f5f8c475f0f46af9121"
format_stamp: "Formatted at 2026-06-19 13:00:36 on dist-test-slave-tpsm"
server_key: "b77bae6f15e2c9c6bd9fb5f1251cb74d"
server_key_iv: "7fe53a4d28fe84d1586a2e60ed784a5b"
server_key_version: "encryptionkey@0"
I20260619 13:00:36.462723 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "aa9070ed926c4f5f8c475f0f46af9121"
format_stamp: "Formatted at 2026-06-19 13:00:36 on dist-test-slave-tpsm"
server_key: "b77bae6f15e2c9c6bd9fb5f1251cb74d"
server_key_iv: "7fe53a4d28fe84d1586a2e60ed784a5b"
server_key_version: "encryptionkey@0"
I20260619 13:00:36.468617 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 13:00:36.474128 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:36.476145 4801 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 13:00:36.476369 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "aa9070ed926c4f5f8c475f0f46af9121"
format_stamp: "Formatted at 2026-06-19 13:00:36 on dist-test-slave-tpsm"
server_key: "b77bae6f15e2c9c6bd9fb5f1251cb74d"
server_key_iv: "7fe53a4d28fe84d1586a2e60ed784a5b"
server_key_version: "encryptionkey@0"
I20260619 13:00:36.476621 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:36.500754 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:36.505036 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:36.505317 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:36.506196 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:36.507479 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:36.507561 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:36.507674 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:36.507736 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:36.541929 4801 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:40385
I20260619 13:00:36.542016 4928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:40385 every 8 connection(s)
I20260619 13:00:36.543205 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:36.545277 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 4801
I20260619 13:00:36.545451 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:36.545979 1373 external_mini_cluster.cc:1428] Setting key 9d5184453fc8e3ec97b59fdb0f369d67
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874036, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:36.564538 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:36.545994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:50935 (local address 127.1.87.126:46479)
0619 13:00:36.546599 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:36.546609 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:36.547470 (+ 861us) server_negotiation.cc:407] Connection header received
0619 13:00:36.549770 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:36.549777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:36.549862 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:36.549978 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:36.551515 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.552281 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.553808 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.554234 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.554368 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:36.557972 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:36.557977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:36.557980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:36.558018 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:36.561119 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.562097 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.562101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.562103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.562241 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.562778 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.562782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.562785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.563079 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:36.563276 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:36.563976 (+ 700us) server_negotiation.cc:299] Negotiation successful
0619 13:00:36.564186 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448}
I20260619 13:00:36.565482 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:36.546453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:46479 (local address 127.1.87.67:50935)
0619 13:00:36.547334 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:36.547382 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:36.549503 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:36.550225 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:36.550234 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:36.550727 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:36.551294 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:36.551368 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.552430 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.552436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:36.553660 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:36.553672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.554454 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.554463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:36.554725 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:36.555459 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:36.555482 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:36.557712 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:36.561280 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:36.561290 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:36.561308 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:36.561922 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:36.562365 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:36.562370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:36.562374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:36.562611 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:36.563249 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:36.563258 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:36.563622 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 13:00:36.563884 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 13:00:36.564419 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":316,"threads_started":1}
I20260619 13:00:36.566969 4929 heartbeater.cc:347] Connected to a master server at 127.1.87.126:46479
I20260619 13:00:36.567356 4929 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:36.568082 4929 heartbeater.cc:511] Master 127.1.87.126:46479 requested a full tablet report, sending...
I20260619 13:00:36.569559 4473 ts_manager.cc:194] Registered new tserver with Master: aa9070ed926c4f5f8c475f0f46af9121 (127.1.87.67:40385)
I20260619 13:00:36.570374 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:50935
I20260619 13:00:36.574087 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:00:36.592654 4662 heartbeater.cc:503] Master 127.1.87.126:46479 was elected leader, sending a full tablet report...
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:36.624475 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:36.610865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.1.87.126:46479)
0619 13:00:36.611039 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:36.611043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:36.611493 (+ 450us) server_negotiation.cc:407] Connection header received
0619 13:00:36.611632 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:36.611636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:36.611703 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:36.611801 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:36.613209 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.614057 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.615383 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.615788 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.615888 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:36.619719 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:36.619724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:36.619727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:36.619764 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:36.622113 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.622711 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.622715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.622717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.622860 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.623172 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.623175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.623177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.623552 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:36.623768 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:36.624002 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 13:00:36.624243 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 13:00:36.624989 4473 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46910
I20260619 13:00:36.626706 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4534
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:36.641284 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:36.627778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.1.87.65:37487)
0619 13:00:36.628412 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:36.628419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:36.628442 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:00:36.628494 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:36.628500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:36.628699 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:36.628823 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:36.629836 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.630831 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.632010 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.632542 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.632640 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:36.636464 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:36.636474 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:36.636488 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:36.636521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:36.639025 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.639676 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.639683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.639685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.639807 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.640101 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.640104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.640106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.640401 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:36.640606 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:36.640833 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 13:00:36.641015 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":342,"threads_started":1}
W20260619 13:00:36.642480 4657 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46338
W20260619 13:00:36.643602 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4534: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:00:36.643790 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 4534
I20260619 13:00:36.659821 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4668
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:00:36.675220 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:36.660637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.1.87.66:33389)
0619 13:00:36.661227 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:36.661235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:36.661261 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:00:36.661319 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:36.661326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:36.661524 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:36.661667 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:36.662842 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.663944 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.665174 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.665712 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.665811 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:36.669795 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:36.669806 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:36.669821 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:36.669863 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:36.672808 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.673443 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.673450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.673453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.673579 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.673949 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.673952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.673954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.674269 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:36.674489 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:36.674742 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 13:00:36.674939 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":337,"threads_started":1}
W20260619 13:00:36.676468 4791 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39680
W20260619 13:00:36.677486 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4668: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:00:36.677624 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 4668
I20260619 13:00:36.692986 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4801
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:00:36.709612 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:36.694010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.1.87.67:40385)
0619 13:00:36.694620 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:36.694627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:36.694656 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:00:36.694712 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:36.694719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:36.694934 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:36.695056 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:36.696626 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.697675 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.698908 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:36.699555 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:36.699662 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:36.704062 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:36.704076 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:36.704095 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:36.704157 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:36.706761 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.707493 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.707500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.707502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.707636 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:36.708090 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:36.708093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:36.708095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:36.708500 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:36.708729 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:36.708998 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 13:00:36.709329 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":329,"threads_started":1}
W20260619 13:00:36.710898 4924 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53508
W20260619 13:00:36.712013 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4801: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:00:36.712159 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 4801
I20260619 13:00:36.727527 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4441
I20260619 13:00:36.728945 4503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:36.832576 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 4441
2026-06-19T13:00:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3551 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4946](info): set up 2 sockets
Jun 19 13:00:36 dist-test-slave-tpsm krb5kdc[4946](info): commencing operation
krb5kdc: starting...
W20260619 13:00:37.927728 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:38 dist-test-slave-tpsm krb5kdc[4946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:38.115749 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:42899
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:42899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:38.444404 4959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:38.444693 4959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:38.444783 4959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:38.452180 4959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:38.452291 4959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:38.452337 4959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:38.452384 4959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:38.452430 4959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:38.462218 4959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:42899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:38.463604 4959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:38.464834 4959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:38.478520 4964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:38.478477 4967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:38.478605 4965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:38.479143 4959 server_base.cc:1034] running on GCE node
I20260619 13:00:38.480038 4959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:38.481350 4959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:38.482549 4959 hybrid_clock.cc:648] HybridClock initialized: now 1781874038482503 us; error 41 us; skew 500 ppm
I20260619 13:00:38.482918 4959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:38 dist-test-slave-tpsm krb5kdc[4946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874038, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:38.486018 4959 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:38.487577 4959 webserver.cc:460] Webserver started at http://127.1.87.126:41943/ using document root <none> and password file <none>
I20260619 13:00:38.488248 4959 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:38.488392 4959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:38.488713 4959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:38.491672 4959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "591cf29aeaa9483aac5f117a31ba212f"
format_stamp: "Formatted at 2026-06-19 13:00:38 on dist-test-slave-tpsm"
server_key: "173f4bcf6c89df34f8c43d41f135267e"
server_key_iv: "3a3ef41328d37619cfac366119b90ff1"
server_key_version: "encryptionkey@0"
I20260619 13:00:38.492458 4959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "591cf29aeaa9483aac5f117a31ba212f"
format_stamp: "Formatted at 2026-06-19 13:00:38 on dist-test-slave-tpsm"
server_key: "173f4bcf6c89df34f8c43d41f135267e"
server_key_iv: "3a3ef41328d37619cfac366119b90ff1"
server_key_version: "encryptionkey@0"
I20260619 13:00:38.497516 4959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:00:38.500881 4974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:38.502144 4959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:00:38.502321 4959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "591cf29aeaa9483aac5f117a31ba212f"
format_stamp: "Formatted at 2026-06-19 13:00:38 on dist-test-slave-tpsm"
server_key: "173f4bcf6c89df34f8c43d41f135267e"
server_key_iv: "3a3ef41328d37619cfac366119b90ff1"
server_key_version: "encryptionkey@0"
I20260619 13:00:38.502516 4959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:38.517239 4959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:38.521740 4959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:38.522039 4959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:38.541287 4959 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:42899
I20260619 13:00:38.541311 5025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:42899 every 8 connection(s)
I20260619 13:00:38.542698 4959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:38.544754 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 4959
I20260619 13:00:38.545002 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:38.545665 1373 external_mini_cluster.cc:1428] Setting key 3d1561e546a3f51ed2ee176bdb1f0c54
I20260619 13:00:38.546805 5026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:00:38 dist-test-slave-tpsm krb5kdc[4946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:38.560741 5026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f: Bootstrap starting.
I20260619 13:00:38.564842 5026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:38.566114 5026 log.cc:824] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:38.569641 5026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f: No bootstrap required, opened a new log
I20260619 13:00:38.570225 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:38.547721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39500 (local address 127.1.87.126:42899)
0619 13:00:38.548527 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:38.548543 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:38.548591 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:00:38.550381 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:38.550409 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:38.550918 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:38.551404 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:38.552761 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:38.554805 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:38.556070 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:38.557091 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:38.557300 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:38.561520 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:38.561531 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:38.561549 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:38.561597 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:38.565652 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:38.566417 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:38.566425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:38.566433 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:38.566661 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:38.567101 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:38.567104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:38.567106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:38.567789 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:38.568061 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:38.568445 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 13:00:38.568883 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":581,"thread_start_us":337,"threads_started":1}
I20260619 13:00:38.577966 5026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591cf29aeaa9483aac5f117a31ba212f" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42899 } }
I20260619 13:00:38.578474 5026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:38.578588 5026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591cf29aeaa9483aac5f117a31ba212f, State: Initialized, Role: FOLLOWER
I20260619 13:00:38.579231 5026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591cf29aeaa9483aac5f117a31ba212f" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42899 } }
I20260619 13:00:38.579495 5026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:38.579664 5026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:38.579829 5026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:38.581923 5026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591cf29aeaa9483aac5f117a31ba212f" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42899 } }
I20260619 13:00:38.582464 5026 leader_election.cc:302] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 591cf29aeaa9483aac5f117a31ba212f; no voters:
I20260619 13:00:38.583067 5026 leader_election.cc:288] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:38.583232 5031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:38.584061 5031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [term 1 LEADER]: Becoming Leader. State: Replica: 591cf29aeaa9483aac5f117a31ba212f, State: Running, Role: LEADER
I20260619 13:00:38.584549 5031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591cf29aeaa9483aac5f117a31ba212f" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42899 } }
I20260619 13:00:38.585162 5026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:38.587538 5033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 591cf29aeaa9483aac5f117a31ba212f. Latest consensus state: current_term: 1 leader_uuid: "591cf29aeaa9483aac5f117a31ba212f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591cf29aeaa9483aac5f117a31ba212f" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42899 } } }
I20260619 13:00:38.587622 5032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "591cf29aeaa9483aac5f117a31ba212f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591cf29aeaa9483aac5f117a31ba212f" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42899 } } }
I20260619 13:00:38.587970 5033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:38.588032 5032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:38.588979 5037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:38.594774 5037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:38.603847 5037 catalog_manager.cc:1347] Generated new cluster ID: 842fe85ce9644ee599b64e5621fd5880
I20260619 13:00:38.603971 5037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:38.618193 5037 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:38.619568 5037 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:38.632532 5037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 591cf29aeaa9483aac5f117a31ba212f: Generated new TSK 0
I20260619 13:00:38.633589 5037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:00:38.686566 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:42899
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:39.015877 5052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:39.016149 5052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:39.016326 5052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:39.023587 5052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:39.023703 5052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:39.023912 5052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:39.033711 5052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:42899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:39.035030 5052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:39.036350 5052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:39.049593 5057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:39.049482 5058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:39.049454 5060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:39.050482 5052 server_base.cc:1034] running on GCE node
I20260619 13:00:39.051192 5052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:39.052187 5052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:39.053428 5052 hybrid_clock.cc:648] HybridClock initialized: now 1781874039053405 us; error 76 us; skew 500 ppm
I20260619 13:00:39.053808 5052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:39 dist-test-slave-tpsm krb5kdc[4946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:39.057281 5052 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:00:39.058830 5052 webserver.cc:460] Webserver started at http://127.1.87.65:35343/ using document root <none> and password file <none>
I20260619 13:00:39.059628 5052 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:39.059774 5052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:39.060086 5052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:39.062963 5052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "58f22bdb32fb413ba99a88b81fd88afb"
format_stamp: "Formatted at 2026-06-19 13:00:39 on dist-test-slave-tpsm"
server_key: "f55bf0117eb36e6746c4f83f70019024"
server_key_iv: "807f38580326da04f0c71e997cde7561"
server_key_version: "encryptionkey@0"
I20260619 13:00:39.063776 5052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "58f22bdb32fb413ba99a88b81fd88afb"
format_stamp: "Formatted at 2026-06-19 13:00:39 on dist-test-slave-tpsm"
server_key: "f55bf0117eb36e6746c4f83f70019024"
server_key_iv: "807f38580326da04f0c71e997cde7561"
server_key_version: "encryptionkey@0"
I20260619 13:00:39.069535 5052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:00:39.074309 5067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:39.076098 5052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:39.076303 5052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "58f22bdb32fb413ba99a88b81fd88afb"
format_stamp: "Formatted at 2026-06-19 13:00:39 on dist-test-slave-tpsm"
server_key: "f55bf0117eb36e6746c4f83f70019024"
server_key_iv: "807f38580326da04f0c71e997cde7561"
server_key_version: "encryptionkey@0"
I20260619 13:00:39.076530 5052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:39.091028 5052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:39.095731 5052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:39.096031 5052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:39.096993 5052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:39.098282 5052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:39.098388 5052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:39.098532 5052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:39.098610 5052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:39.132386 5052 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:44843
I20260619 13:00:39.132417 5179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:44843 every 8 connection(s)
I20260619 13:00:39.133755 5052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:39.135591 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 5052
I20260619 13:00:39.135834 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:39.136655 1373 external_mini_cluster.cc:1428] Setting key df71da3b5499444d6ceed2155a2bba0e
Jun 19 13:00:39 dist-test-slave-tpsm krb5kdc[4946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
I20260619 13:00:39.157075 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:39.136635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:33357 (local address 127.1.87.126:42899)
0619 13:00:39.137134 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:39.137139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:39.138239 (+ 1100us) server_negotiation.cc:407] Connection header received
0619 13:00:39.140323 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:39.140331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:39.140417 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:39.140537 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:39.141986 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:39.143124 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:39.145574 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:39.145991 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:39.146105 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:39.150729 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:39.150735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:39.150738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:39.150769 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:39.153486 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:39.154482 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:39.154486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:39.154488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:39.154600 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:39.155111 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:39.155114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:39.155116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:39.155478 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:39.155588 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:39.156636 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0619 13:00:39.156826 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":271,"threads_started":1}
I20260619 13:00:39.158267 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:39.137185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42899 (local address 127.1.87.65:33357)
0619 13:00:39.138056 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:39.138109 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:39.140050 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:39.140738 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:39.140746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:39.141244 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:39.141805 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:39.141823 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:39.143357 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:39.143366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:39.145401 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:39.145418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:39.146221 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:39.146230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:39.146549 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:39.147289 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:39.147347 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:39.150462 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:39.153646 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:39.153658 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:39.153679 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:39.154347 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:39.154715 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:39.154721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:39.154725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:39.154993 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:39.155614 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:39.155625 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:39.156148 (+ 523us) client_negotiation.cc:769] Sending connection context
0619 13:00:39.156475 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 13:00:39.157030 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":319,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
I20260619 13:00:39.160202 5180 heartbeater.cc:347] Connected to a master server at 127.1.87.126:42899
I20260619 13:00:39.160547 5180 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:39.161435 5180 heartbeater.cc:511] Master 127.1.87.126:42899 requested a full tablet report, sending...
I20260619 13:00:39.163686 4991 ts_manager.cc:194] Registered new tserver with Master: 58f22bdb32fb413ba99a88b81fd88afb (127.1.87.65:44843)
I20260619 13:00:39.165652 4991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:33357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:00:39.182610 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:42899
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:39.501268 5186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:39.501564 5186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:39.501734 5186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:39.509126 5186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:39.509244 5186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:39.509479 5186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:39.519162 5186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:42899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:39.520476 5186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:39.521739 5186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:39.534509 5194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:39.534497 5192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:39.534838 5191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:39.535264 5186 server_base.cc:1034] running on GCE node
I20260619 13:00:39.535934 5186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:39.537523 5186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:39.538882 5186 hybrid_clock.cc:648] HybridClock initialized: now 1781874039538810 us; error 56 us; skew 500 ppm
I20260619 13:00:39.539240 5186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:39 dist-test-slave-tpsm krb5kdc[4946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:39.542989 5186 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:00:39.551259 5186 webserver.cc:460] Webserver started at http://127.1.87.66:43151/ using document root <none> and password file <none>
I20260619 13:00:39.552047 5186 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:39.552183 5186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:39.552513 5186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:39.555507 5186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "0c07a4ad662042ce9266760bbba0c2ed"
format_stamp: "Formatted at 2026-06-19 13:00:39 on dist-test-slave-tpsm"
server_key: "a0da1ec055bec4f0df2df7248a94b3c3"
server_key_iv: "8bd2c1757f9f72a0e03b6af5d3c6f499"
server_key_version: "encryptionkey@0"
I20260619 13:00:39.556308 5186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "0c07a4ad662042ce9266760bbba0c2ed"
format_stamp: "Formatted at 2026-06-19 13:00:39 on dist-test-slave-tpsm"
server_key: "a0da1ec055bec4f0df2df7248a94b3c3"
server_key_iv: "8bd2c1757f9f72a0e03b6af5d3c6f499"
server_key_version: "encryptionkey@0"
I20260619 13:00:39.561594 5186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:00:39.565114 5201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:39.566579 5186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:00:39.566771 5186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "0c07a4ad662042ce9266760bbba0c2ed"
format_stamp: "Formatted at 2026-06-19 13:00:39 on dist-test-slave-tpsm"
server_key: "a0da1ec055bec4f0df2df7248a94b3c3"
server_key_iv: "8bd2c1757f9f72a0e03b6af5d3c6f499"
server_key_version: "encryptionkey@0"
I20260619 13:00:39.566993 5186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:39.587610 5186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:39.592185 5186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:39.592476 5186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:39.593436 5186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:39.595186 5186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:39.595330 5186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:39.595489 5186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:39.595566 5186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:39.633076 5186 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:33857
I20260619 13:00:39.633100 5313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:33857 every 8 connection(s)
I20260619 13:00:39.634439 5186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:39.641991 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 5186
I20260619 13:00:39.642230 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:39.642652 1373 external_mini_cluster.cc:1428] Setting key 8af034ea7f94eedaf507dd0ea0be99e9
Jun 19 13:00:39 dist-test-slave-tpsm krb5kdc[4946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:39.658365 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:39.637110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:47795 (local address 127.1.87.126:42899)
0619 13:00:39.637312 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:39.637318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:39.638463 (+ 1145us) server_negotiation.cc:407] Connection header received
0619 13:00:39.640944 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:39.640950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:39.641049 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:39.641186 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:39.642851 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:39.643722 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:39.645971 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:39.646390 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:39.646496 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:39.650795 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:39.650804 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:39.650810 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:39.650857 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:39.654453 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:39.655266 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:39.655274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:39.655281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:39.655466 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:39.656264 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:39.656272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:39.656278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:39.656682 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:39.656831 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:39.657623 (+ 792us) server_negotiation.cc:299] Negotiation successful
0619 13:00:39.657894 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:00:39.659297 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:39.637586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42899 (local address 127.1.87.66:47795)
0619 13:00:39.638292 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:39.638331 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:39.640559 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:39.641409 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:39.641418 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:39.641934 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:39.642622 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:39.642641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:39.643871 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:39.643877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:39.645766 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:39.645785 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:39.646633 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:39.646643 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:39.647003 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:39.647774 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:39.647814 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:39.650519 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:39.654630 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:39.654638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:39.654653 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:39.655125 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:39.655683 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:39.655697 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:39.655704 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:39.656120 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:39.656857 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:39.656864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:39.657212 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 13:00:39.657479 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 13:00:39.657888 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":293,"threads_started":1}
I20260619 13:00:39.660924 5314 heartbeater.cc:347] Connected to a master server at 127.1.87.126:42899
I20260619 13:00:39.661244 5314 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:39.661948 5314 heartbeater.cc:511] Master 127.1.87.126:42899 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:00:39.663496 4991 ts_manager.cc:194] Registered new tserver with Master: 0c07a4ad662042ce9266760bbba0c2ed (127.1.87.66:33857)
I20260619 13:00:39.664361 4991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:47795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:00:39.688500 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:42899
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:40.007156 5319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:40.007491 5319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:40.007681 5319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:40.014734 5319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:40.014830 5319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:40.015031 5319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:40.024741 5319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:42899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:40.025974 5319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:40.027387 5319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:40.039808 5325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:40.039988 5324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:40.039939 5327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:40.040994 5319 server_base.cc:1034] running on GCE node
I20260619 13:00:40.041639 5319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:40.042502 5319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:40.043721 5319 hybrid_clock.cc:648] HybridClock initialized: now 1781874040043677 us; error 38 us; skew 500 ppm
I20260619 13:00:40.044041 5319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[4946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:40.051481 5319 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:00:40.052992 5319 webserver.cc:460] Webserver started at http://127.1.87.67:42303/ using document root <none> and password file <none>
I20260619 13:00:40.053674 5319 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:40.053817 5319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:40.054123 5319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:40.056970 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "0e43c01c17134a339fcc30eddbf7fc78"
format_stamp: "Formatted at 2026-06-19 13:00:40 on dist-test-slave-tpsm"
server_key: "1ec6af8a8af8f8c8333bb46b78042576"
server_key_iv: "6a92966f3191ebeb84a3256c66f668ef"
server_key_version: "encryptionkey@0"
I20260619 13:00:40.057716 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "0e43c01c17134a339fcc30eddbf7fc78"
format_stamp: "Formatted at 2026-06-19 13:00:40 on dist-test-slave-tpsm"
server_key: "1ec6af8a8af8f8c8333bb46b78042576"
server_key_iv: "6a92966f3191ebeb84a3256c66f668ef"
server_key_version: "encryptionkey@0"
I20260619 13:00:40.062512 5319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:00:40.065790 5334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:40.067135 5319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:40.067366 5319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "0e43c01c17134a339fcc30eddbf7fc78"
format_stamp: "Formatted at 2026-06-19 13:00:40 on dist-test-slave-tpsm"
server_key: "1ec6af8a8af8f8c8333bb46b78042576"
server_key_iv: "6a92966f3191ebeb84a3256c66f668ef"
server_key_version: "encryptionkey@0"
I20260619 13:00:40.067580 5319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:40.092761 5319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:40.096982 5319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:40.097288 5319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:40.098241 5319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:40.099536 5319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:40.099640 5319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:40.099792 5319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:40.099864 5319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:40.135644 5319 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:40367
I20260619 13:00:40.135701 5446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:40367 every 8 connection(s)
I20260619 13:00:40.137054 5319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:40.138114 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 5319
I20260619 13:00:40.138322 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:40.138837 1373 external_mini_cluster.cc:1428] Setting key 34ec85a0a0d2d2e219119e41522e0f5c
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[4946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:40.157194 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:40.139971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:37413 (local address 127.1.87.126:42899)
0619 13:00:40.140159 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:40.140166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:40.141550 (+ 1384us) server_negotiation.cc:407] Connection header received
0619 13:00:40.143546 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:40.143550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:40.143619 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:40.143712 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:40.145071 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.145851 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.147409 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.147837 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.147952 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:40.151479 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:40.151483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:40.151486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:40.151517 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:40.154553 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.155261 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.155265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.155267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.155441 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.155800 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.155803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.155805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.156060 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:40.156166 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:40.156802 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 13:00:40.156955 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:00:40.158190 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:40.140548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:42899 (local address 127.1.87.67:37413)
0619 13:00:40.141366 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:40.141419 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:40.143237 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:40.143900 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:40.143907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:40.144378 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:40.144907 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:40.144922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.145977 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.145981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:40.147141 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:40.147151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.148058 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.148064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:40.148260 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:40.149072 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:40.149095 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:40.151208 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:40.154717 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:40.154725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:40.154746 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:40.155159 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:40.155534 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:40.155537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:40.155539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:40.155708 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:40.156165 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:40.156170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:40.156477 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 13:00:40.156702 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 13:00:40.157081 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":323,"threads_started":1}
I20260619 13:00:40.160197 5447 heartbeater.cc:347] Connected to a master server at 127.1.87.126:42899
I20260619 13:00:40.160538 5447 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:40.161242 5447 heartbeater.cc:511] Master 127.1.87.126:42899 requested a full tablet report, sending...
I20260619 13:00:40.162822 4991 ts_manager.cc:194] Registered new tserver with Master: 0e43c01c17134a339fcc30eddbf7fc78 (127.1.87.67:40367)
I20260619 13:00:40.164024 4991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:37413
I20260619 13:00:40.167398 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:00:40.168551 5180 heartbeater.cc:503] Master 127.1.87.126:42899 was elected leader, sending a full tablet report...
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[4946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[4946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:40.217525 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:40.202943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.1.87.65:44843)
0619 13:00:40.203571 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:40.203578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:40.203623 (+ 45us) server_negotiation.cc:407] Connection header received
0619 13:00:40.203696 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:40.203703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:40.203913 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:40.204035 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:40.205517 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.206702 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.208111 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.208646 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.208758 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:40.212605 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:40.212616 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:40.212630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:40.212664 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:40.215240 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.215856 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.215862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.215864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.215971 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.216275 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.216278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.216280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.216575 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:40.216793 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:40.217074 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 13:00:40.217251 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":297,"threads_started":1}
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[4946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:40.233639 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:40.221520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.1.87.126:42899)
0619 13:00:40.221750 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:40.221758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:40.221864 (+ 106us) server_negotiation.cc:407] Connection header received
0619 13:00:40.221982 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:40.221985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:40.222063 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:40.222140 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:40.223380 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.224068 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.225207 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.225573 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.225661 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:40.229179 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:40.229185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:40.229188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:40.229217 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:40.231522 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.232047 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.232050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.232052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.232168 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.232460 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.232462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.232464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.232751 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:40.232921 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:40.233157 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 13:00:40.233304 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 13:00:40.234179 4991 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39510
I20260619 13:00:40.235153 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 5052
I20260619 13:00:40.247552 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:40.236141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.1.87.65:44843)
0619 13:00:40.236353 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:40.236360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:40.236514 (+ 154us) server_negotiation.cc:407] Connection header received
0619 13:00:40.236689 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:40.236694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:40.236767 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:40.236866 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:40.237976 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.238629 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.239886 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.240318 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.240445 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:40.242618 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:40.242624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:40.242629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:40.242669 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:40.245005 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.245594 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.245600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.245603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.245727 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.246083 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.246088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.246091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.246336 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:40.246573 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:40.246871 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 13:00:40.247084 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 13:00:40.248147 5175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:40.331421 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 5052
I20260619 13:00:40.348501 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 5186
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[4946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:00:40.364113 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:40.349509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.1.87.66:33857)
0619 13:00:40.350029 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:40.350036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:40.350060 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:00:40.350121 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:40.350129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:40.350349 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:40.350538 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:40.351740 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.352743 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.353969 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.354502 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.354643 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:40.358683 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:40.358695 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:40.358710 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:40.358766 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:40.361394 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.361980 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.361986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.361988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.362097 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.362462 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.362467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.362469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.362844 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:40.363098 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:40.363617 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 13:00:40.363827 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":261,"threads_started":1}
I20260619 13:00:40.365340 5309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:40.448444 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 5186
I20260619 13:00:40.464861 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 5319
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[4946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:00:40.483194 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:40.465933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.1.87.67:40367)
0619 13:00:40.466622 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:40.466629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:40.466673 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:00:40.466749 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:40.466773 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:40.467103 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:40.467284 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:40.468536 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.469541 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.470905 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:40.471925 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:40.472050 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:40.477661 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:40.477673 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:40.477688 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:40.477726 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:40.480639 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.481286 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.481293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.481295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.481435 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:40.481758 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:40.481761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:40.481763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:40.482097 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:40.482405 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:40.482568 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 13:00:40.482826 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":249,"threads_started":1}
I20260619 13:00:40.484480 5442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:40.571141 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 5319
I20260619 13:00:40.587069 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 4959
I20260619 13:00:40.588966 5021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:40.693419 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 4959
2026-06-19T13:00:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3860 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[5466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[5466](info): set up 2 sockets
Jun 19 13:00:40 dist-test-slave-tpsm krb5kdc[5466](info): commencing operation
krb5kdc: starting...
W20260619 13:00:41.779762 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:41 dist-test-slave-tpsm krb5kdc[5466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:41.971060 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:36995
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:36995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:42.302378 5479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:42.302728 5479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:42.302886 5479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:42.310040 5479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:42.310190 5479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:42.310308 5479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:42.310410 5479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:42.310504 5479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:42.320221 5479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:36995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:36995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:42.321540 5479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:42.322850 5479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:42.334182 5487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:42.334290 5484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:42.334290 5485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:42.335078 5479 server_base.cc:1034] running on GCE node
I20260619 13:00:42.335892 5479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:42.337188 5479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:42.338387 5479 hybrid_clock.cc:648] HybridClock initialized: now 1781874042338352 us; error 31 us; skew 500 ppm
I20260619 13:00:42.338666 5479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:42 dist-test-slave-tpsm krb5kdc[5466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874042, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:42.341629 5479 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:42.342743 5479 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:42.342856 5479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:42.343190 5479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:42.346112 5479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "e15bd846fd354741971125dbec4e14c4"
format_stamp: "Formatted at 2026-06-19 13:00:42 on dist-test-slave-tpsm"
server_key: "1ca80e7b4876eca58e4574f1c1011995"
server_key_iv: "8e9c80a7b92ef511faf0b3b89b38a43d"
server_key_version: "encryptionkey@0"
I20260619 13:00:42.346892 5479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "e15bd846fd354741971125dbec4e14c4"
format_stamp: "Formatted at 2026-06-19 13:00:42 on dist-test-slave-tpsm"
server_key: "1ca80e7b4876eca58e4574f1c1011995"
server_key_iv: "8e9c80a7b92ef511faf0b3b89b38a43d"
server_key_version: "encryptionkey@0"
I20260619 13:00:42.351706 5479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260619 13:00:42.354817 5493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:42.356087 5479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:42.356257 5479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "e15bd846fd354741971125dbec4e14c4"
format_stamp: "Formatted at 2026-06-19 13:00:42 on dist-test-slave-tpsm"
server_key: "1ca80e7b4876eca58e4574f1c1011995"
server_key_iv: "8e9c80a7b92ef511faf0b3b89b38a43d"
server_key_version: "encryptionkey@0"
I20260619 13:00:42.356463 5479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:42.374862 5479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:42.378690 5479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:42.378947 5479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:42.396334 5479 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:36995
I20260619 13:00:42.396353 5544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:36995 every 8 connection(s)
I20260619 13:00:42.397215 5479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:42.399122 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 5479
I20260619 13:00:42.399384 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:42.400002 1373 external_mini_cluster.cc:1428] Setting key 36822451625cc68fa46f5edbeb2b33bf
I20260619 13:00:42.400884 5545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:00:42 dist-test-slave-tpsm krb5kdc[5466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:42.414256 5545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4: Bootstrap starting.
I20260619 13:00:42.418215 5545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:42.419411 5545 log.cc:824] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:42.422448 5545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4: No bootstrap required, opened a new log
I20260619 13:00:42.422734 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:42.402027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.1.87.126:36995)
0619 13:00:42.402828 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:42.402842 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:42.402888 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:00:42.404515 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:42.404541 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:42.405026 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:42.405428 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:42.406696 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:42.408695 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:42.409956 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:42.410935 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:42.411118 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:42.414797 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:42.414806 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:42.414823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:42.414867 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:42.418642 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:42.419269 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:42.419276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:42.419284 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:42.419493 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:42.419857 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:42.419865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:42.419871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:42.420492 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:42.420795 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:42.421136 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 13:00:42.421510 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":329,"threads_started":1}
I20260619 13:00:42.430344 5545 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15bd846fd354741971125dbec4e14c4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 36995 } }
I20260619 13:00:42.430691 5545 raft_consensus.cc:374] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:42.430763 5545 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15bd846fd354741971125dbec4e14c4, State: Initialized, Role: FOLLOWER
I20260619 13:00:42.431293 5545 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15bd846fd354741971125dbec4e14c4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 36995 } }
I20260619 13:00:42.431541 5545 raft_consensus.cc:388] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:42.431643 5545 raft_consensus.cc:482] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:42.431759 5545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:42.433776 5545 raft_consensus.cc:504] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15bd846fd354741971125dbec4e14c4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 36995 } }
I20260619 13:00:42.434206 5545 leader_election.cc:302] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e15bd846fd354741971125dbec4e14c4; no voters:
I20260619 13:00:42.434697 5545 leader_election.cc:288] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:42.434875 5550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:42.435582 5550 raft_consensus.cc:686] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [term 1 LEADER]: Becoming Leader. State: Replica: e15bd846fd354741971125dbec4e14c4, State: Running, Role: LEADER
I20260619 13:00:42.436167 5550 consensus_queue.cc:237] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15bd846fd354741971125dbec4e14c4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 36995 } }
I20260619 13:00:42.436902 5545 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:42.439282 5551 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e15bd846fd354741971125dbec4e14c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15bd846fd354741971125dbec4e14c4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 36995 } } }
I20260619 13:00:42.439378 5552 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15bd846fd354741971125dbec4e14c4. Latest consensus state: current_term: 1 leader_uuid: "e15bd846fd354741971125dbec4e14c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15bd846fd354741971125dbec4e14c4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 36995 } } }
I20260619 13:00:42.439837 5552 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:42.440833 5551 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:42.444168 5559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:42.448810 5559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:42.456590 5559 catalog_manager.cc:1347] Generated new cluster ID: dcb5507b50954acfb9965ca2cec419b8
I20260619 13:00:42.456714 5559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:42.478259 5559 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:42.479579 5559 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:42.489661 5559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4: Generated new TSK 0
I20260619 13:00:42.490609 5559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:00:42.543823 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:36995
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:42.869892 5571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:42.870147 5571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:42.870301 5571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:42.877555 5571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:42.877662 5571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:42.877782 5571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:42.887305 5571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:36995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:42.888536 5571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:42.889755 5571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:42.901747 5576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:42.901768 5577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:42.901803 5579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:42.903343 5571 server_base.cc:1034] running on GCE node
I20260619 13:00:42.903999 5571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:42.904902 5571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:42.906123 5571 hybrid_clock.cc:648] HybridClock initialized: now 1781874042906081 us; error 58 us; skew 500 ppm
I20260619 13:00:42.906430 5571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:42 dist-test-slave-tpsm krb5kdc[5466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874042, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:42.909420 5571 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:00:42.910494 5571 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:42.910626 5571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:42.910929 5571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:42.913671 5571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "f4a62c7ec8374ddea7ce178377f278e5"
format_stamp: "Formatted at 2026-06-19 13:00:42 on dist-test-slave-tpsm"
server_key: "4ffa894a9a34af727cb8d14da7be5650"
server_key_iv: "89a702773ed0fea821d73dcc8fb0c122"
server_key_version: "encryptionkey@0"
I20260619 13:00:42.914456 5571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "f4a62c7ec8374ddea7ce178377f278e5"
format_stamp: "Formatted at 2026-06-19 13:00:42 on dist-test-slave-tpsm"
server_key: "4ffa894a9a34af727cb8d14da7be5650"
server_key_iv: "89a702773ed0fea821d73dcc8fb0c122"
server_key_version: "encryptionkey@0"
I20260619 13:00:42.919111 5571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 13:00:42.922230 5585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:42.923511 5571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:00:42.923682 5571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "f4a62c7ec8374ddea7ce178377f278e5"
format_stamp: "Formatted at 2026-06-19 13:00:42 on dist-test-slave-tpsm"
server_key: "4ffa894a9a34af727cb8d14da7be5650"
server_key_iv: "89a702773ed0fea821d73dcc8fb0c122"
server_key_version: "encryptionkey@0"
I20260619 13:00:42.923900 5571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:42.949579 5571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:42.953647 5571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:42.953941 5571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:42.954891 5571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:42.956260 5571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:42.956390 5571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:42.956512 5571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:42.956597 5571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:42.989713 5571 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:41759
I20260619 13:00:42.989800 5697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:41759 every 8 connection(s)
I20260619 13:00:42.990545 5571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:42.992710 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 5571
I20260619 13:00:42.992924 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:42.993691 1373 external_mini_cluster.cc:1428] Setting key 65d0a360b01e85585692fb678d947c7a
Jun 19 13:00:43 dist-test-slave-tpsm krb5kdc[5466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874042, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:43.011932 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:42.992879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:33387 (local address 127.1.87.126:36995)
0619 13:00:42.996618 (+ 3739us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:42.996625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:42.996647 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:00:42.997933 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:42.997938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:42.998023 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:42.998139 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:42.999609 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.000297 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.001841 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.002224 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.002357 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:43.006406 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:43.006411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:43.006414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:43.006443 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:43.008992 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:43.009727 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:43.009731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:43.009733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:43.009842 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:43.010249 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:43.010254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:43.010257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:43.010505 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:43.010655 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:43.011351 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 13:00:43.011560 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3612,"thread_start_us":3522,"threads_started":1}
I20260619 13:00:43.012840 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:42.993520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:36995 (local address 127.1.87.65:33387)
0619 13:00:42.995895 (+ 2375us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:42.995946 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:42.997669 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:42.998356 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:42.998363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:42.998854 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:42.999455 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:42.999472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.000446 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.000451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:43.001684 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:43.001696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.002470 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.002478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:43.002804 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:43.003637 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:43.003670 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:43.006187 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:43.009140 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:43.009148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:43.009164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:43.009585 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:43.009949 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:43.009955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:43.009958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:43.010132 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:43.010631 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:43.010639 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:43.010956 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 13:00:43.011242 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 13:00:43.011748 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2087,"thread_start_us":1894,"threads_started":1}
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
I20260619 13:00:43.014323 5698 heartbeater.cc:347] Connected to a master server at 127.1.87.126:36995
I20260619 13:00:43.014604 5698 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:43.015168 5698 heartbeater.cc:511] Master 127.1.87.126:36995 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
I20260619 13:00:43.017385 5510 ts_manager.cc:194] Registered new tserver with Master: f4a62c7ec8374ddea7ce178377f278e5 (127.1.87.65:41759)
I20260619 13:00:43.019495 5510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:33387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:00:43.038815 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:36995
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:43.355825 5704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:43.356065 5704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:43.356158 5704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:43.363058 5704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:43.363154 5704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:43.363270 5704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:43.372473 5704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:36995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:43.373559 5704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:43.374769 5704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:43.386919 5710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:43.386940 5709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:43.387405 5712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:43.387888 5704 server_base.cc:1034] running on GCE node
I20260619 13:00:43.388381 5704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:43.389158 5704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:43.390415 5704 hybrid_clock.cc:648] HybridClock initialized: now 1781874043390376 us; error 61 us; skew 500 ppm
I20260619 13:00:43.390746 5704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:43 dist-test-slave-tpsm krb5kdc[5466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874043, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:43.393658 5704 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:00:43.394681 5704 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:43.394807 5704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:43.395110 5704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:43.397792 5704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "d33ffd20763b4d5d8fbc793e862a2023"
format_stamp: "Formatted at 2026-06-19 13:00:43 on dist-test-slave-tpsm"
server_key: "7c62b5776eb4cebce7f73390838172c9"
server_key_iv: "3facedb3bb5a34378a852caa967d9437"
server_key_version: "encryptionkey@0"
I20260619 13:00:43.398563 5704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "d33ffd20763b4d5d8fbc793e862a2023"
format_stamp: "Formatted at 2026-06-19 13:00:43 on dist-test-slave-tpsm"
server_key: "7c62b5776eb4cebce7f73390838172c9"
server_key_iv: "3facedb3bb5a34378a852caa967d9437"
server_key_version: "encryptionkey@0"
I20260619 13:00:43.403223 5704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 13:00:43.406554 5718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:43.407866 5704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:43.408037 5704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "d33ffd20763b4d5d8fbc793e862a2023"
format_stamp: "Formatted at 2026-06-19 13:00:43 on dist-test-slave-tpsm"
server_key: "7c62b5776eb4cebce7f73390838172c9"
server_key_iv: "3facedb3bb5a34378a852caa967d9437"
server_key_version: "encryptionkey@0"
I20260619 13:00:43.408236 5704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:43.422864 5704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:43.426318 5704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:43.426582 5704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:43.427578 5704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:43.428854 5704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:43.428972 5704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:43.429107 5704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:43.429201 5704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:43.461293 5704 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:43289
I20260619 13:00:43.461374 5830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:43289 every 8 connection(s)
I20260619 13:00:43.462131 5704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:43.467104 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 5704
I20260619 13:00:43.467289 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:43.467849 1373 external_mini_cluster.cc:1428] Setting key 56489f5d449ee496cddd19baa9ab58e3
Jun 19 13:00:43 dist-test-slave-tpsm krb5kdc[5466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874043, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:43.480489 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:43.464577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:49375 (local address 127.1.87.126:36995)
0619 13:00:43.464790 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:43.464796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:43.465810 (+ 1014us) server_negotiation.cc:407] Connection header received
0619 13:00:43.467643 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:43.467649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:43.467730 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:43.467839 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:43.469226 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.469962 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.471499 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.471912 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.472046 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:43.475259 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:43.475264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:43.475267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:43.475297 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:43.477674 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:43.478367 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:43.478375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:43.478383 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:43.478502 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:43.478899 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:43.478906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:43.478909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:43.479160 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:43.479316 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:43.479945 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 13:00:43.480152 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:00:43.481483 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:43.464996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:36995 (local address 127.1.87.66:49375)
0619 13:00:43.465652 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:43.465696 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:43.467384 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:43.468059 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:43.468067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:43.468557 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:43.469063 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:43.469078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.470114 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.470124 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:43.471308 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:43.471371 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.472130 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.472136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:43.472355 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:43.473029 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:43.473054 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:43.475040 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:43.477822 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:43.477830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:43.477844 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:43.478244 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:43.478612 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:43.478618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:43.478622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:43.478792 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:43.479290 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:43.479297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:43.479618 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 13:00:43.479873 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 13:00:43.480374 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":270,"threads_started":1}
I20260619 13:00:43.482889 5831 heartbeater.cc:347] Connected to a master server at 127.1.87.126:36995
I20260619 13:00:43.483168 5831 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:43.483837 5831 heartbeater.cc:511] Master 127.1.87.126:36995 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:00:43.485232 5510 ts_manager.cc:194] Registered new tserver with Master: d33ffd20763b4d5d8fbc793e862a2023 (127.1.87.66:43289)
I20260619 13:00:43.486001 5510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:49375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:00:43.508430 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:36995
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:43.825052 5836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:43.825328 5836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:43.825495 5836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:43.832929 5836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:43.833051 5836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:43.833248 5836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:43.843359 5836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:36995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:43.844724 5836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:43.846031 5836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:43.859791 5841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:43.859736 5842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:43.859825 5844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:43.860600 5836 server_base.cc:1034] running on GCE node
I20260619 13:00:43.861233 5836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:43.862136 5836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:43.863436 5836 hybrid_clock.cc:648] HybridClock initialized: now 1781874043863383 us; error 50 us; skew 500 ppm
I20260619 13:00:43.863787 5836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:43 dist-test-slave-tpsm krb5kdc[5466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874043, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:43.867285 5836 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:00:43.868963 5836 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:43.869105 5836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:43.869453 5836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:43.872494 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "aff7556c661a47cfb3613fc21a331924"
format_stamp: "Formatted at 2026-06-19 13:00:43 on dist-test-slave-tpsm"
server_key: "5888cd6d19b576a54bbb64d86f051faf"
server_key_iv: "776066fb597412075beaa74126c0af5f"
server_key_version: "encryptionkey@0"
I20260619 13:00:43.873323 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "aff7556c661a47cfb3613fc21a331924"
format_stamp: "Formatted at 2026-06-19 13:00:43 on dist-test-slave-tpsm"
server_key: "5888cd6d19b576a54bbb64d86f051faf"
server_key_iv: "776066fb597412075beaa74126c0af5f"
server_key_version: "encryptionkey@0"
I20260619 13:00:43.878491 5836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:00:43.881944 5850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:43.883234 5836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:43.883461 5836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "aff7556c661a47cfb3613fc21a331924"
format_stamp: "Formatted at 2026-06-19 13:00:43 on dist-test-slave-tpsm"
server_key: "5888cd6d19b576a54bbb64d86f051faf"
server_key_iv: "776066fb597412075beaa74126c0af5f"
server_key_version: "encryptionkey@0"
I20260619 13:00:43.883699 5836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:43.899178 5836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:43.902971 5836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:43.903247 5836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:43.904201 5836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:43.905599 5836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:43.905719 5836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:43.905853 5836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:43.905936 5836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:43.938841 5836 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:39969
I20260619 13:00:43.938925 5962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:39969 every 8 connection(s)
I20260619 13:00:43.939704 5836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:43.946548 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 5836
I20260619 13:00:43.946748 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:43.947283 1373 external_mini_cluster.cc:1428] Setting key 72a2e747339f5c8f61914ef2452f3585
Jun 19 13:00:43 dist-test-slave-tpsm krb5kdc[5466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874043, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:43.959239 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:43.942015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:37963 (local address 127.1.87.126:36995)
0619 13:00:43.942197 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:43.942202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:43.943330 (+ 1128us) server_negotiation.cc:407] Connection header received
0619 13:00:43.945066 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:43.945071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:43.945188 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:43.945301 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:43.946756 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.947861 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.949971 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.950441 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.950535 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:43.953743 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:43.953748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:43.953751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:43.953781 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:43.956477 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:43.957254 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:43.957258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:43.957260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:43.957386 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:43.957782 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:43.957785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:43.957787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:43.958040 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:43.958159 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:43.958811 (+ 652us) server_negotiation.cc:299] Negotiation successful
0619 13:00:43.958968 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:00:43.960378 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:43.942484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:36995 (local address 127.1.87.67:37963)
0619 13:00:43.943139 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:43.943195 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:43.944806 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:43.945519 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:43.945526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:43.946016 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:43.946538 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:43.946553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.948005 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.948017 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:43.949813 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:43.949826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.950626 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.950631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:43.950805 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:43.951438 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:43.951463 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:43.953543 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:43.956648 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:43.956654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:43.956668 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:43.957120 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:43.957504 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:43.957509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:43.957511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:43.957686 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:43.958173 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:43.958185 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:43.958494 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 13:00:43.958729 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 13:00:43.959107 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":276,"threads_started":1}
I20260619 13:00:43.961901 5963 heartbeater.cc:347] Connected to a master server at 127.1.87.126:36995
I20260619 13:00:43.962245 5963 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:43.962954 5963 heartbeater.cc:511] Master 127.1.87.126:36995 requested a full tablet report, sending...
I20260619 13:00:43.964385 5509 ts_manager.cc:194] Registered new tserver with Master: aff7556c661a47cfb3613fc21a331924 (127.1.87.67:39969)
I20260619 13:00:43.965261 5509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:37963
I20260619 13:00:43.974745 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:00:43.991531 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:43.979129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.1.87.126:36995)
0619 13:00:43.979283 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:43.979287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:43.979787 (+ 500us) server_negotiation.cc:407] Connection header received
0619 13:00:43.979984 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:43.979989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:43.980062 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:43.980165 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:43.981529 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.982255 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.983631 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:43.984047 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:43.984180 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:43.986290 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:43.986295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:43.986300 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:43.986336 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:43.988944 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:43.989673 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:43.989680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:43.989684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:43.989804 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:43.990159 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:43.990165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:43.990169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:43.990459 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:43.990725 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:43.991023 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 13:00:43.991220 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:00:43.995656 5509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:00:44.002254 5509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:44.023079 5698 heartbeater.cc:503] Master 127.1.87.126:36995 was elected leader, sending a full tablet report...
I20260619 13:00:44.036659 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.014369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.1.87.66:43289)
0619 13:00:44.026796 (+ 12427us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:44.026808 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:44.027804 (+ 996us) server_negotiation.cc:407] Connection header received
0619 13:00:44.028223 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:44.028231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:44.028463 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:44.028633 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:44.029791 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.031692 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.034111 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.035355 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.035477 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.035691 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:44.036151 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 13:00:44.036330 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12189,"thread_start_us":12440,"threads_started":1}
I20260619 13:00:44.036736 5976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.013606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:43289 (local address 127.0.0.1:42742)
0619 13:00:44.027635 (+ 14029us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:44.027689 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:44.027974 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:44.028891 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:44.028899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:44.028939 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:44.029611 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.029626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.031869 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.031875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.033954 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.033965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.035802 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.035806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.035927 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.035957 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:00:44.036096 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 13:00:44.036291 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13724,"mutex_wait_us":144,"spinlock_wait_cycles":24960,"thread_start_us":13573,"threads_started":1}
I20260619 13:00:44.037894 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.013621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:41759 (local address 127.0.0.1:51434)
0619 13:00:44.027667 (+ 14046us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:44.027688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:44.027974 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:44.028891 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:44.028899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:44.028943 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:44.030145 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.030157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.031922 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.031927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.034507 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.034521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.036621 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.036627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.037196 (+ 569us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.037230 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:00:44.037319 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 13:00:44.037443 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13930,"mutex_wait_us":748,"spinlock_wait_cycles":21376,"thread_start_us":13810,"threads_started":1}
I20260619 13:00:44.037993 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.029089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:39969 (local address 127.0.0.1:35590)
0619 13:00:44.029739 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:44.029762 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:44.029899 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:44.030505 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:44.030510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:44.030534 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:44.030911 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.030922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.032576 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.032581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.034766 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.034777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.037147 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.037154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.037376 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.037404 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 13:00:44.037475 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 13:00:44.037601 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":4,"thread_start_us":384,"threads_started":1}
I20260619 13:00:44.038769 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.013746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.1.87.65:41759)
0619 13:00:44.027116 (+ 13370us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:44.027125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:44.027828 (+ 703us) server_negotiation.cc:407] Connection header received
0619 13:00:44.028210 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:44.028218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:44.028485 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:44.028644 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:44.030326 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.031754 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.034693 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.036133 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.036261 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.036467 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:44.038275 (+ 1808us) server_negotiation.cc:299] Negotiation successful
0619 13:00:44.038430 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13193,"thread_start_us":13371,"threads_started":1}
I20260619 13:00:44.039950 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.028515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.1.87.67:39969)
0619 13:00:44.029322 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:44.029331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:44.029868 (+ 537us) server_negotiation.cc:407] Connection header received
0619 13:00:44.030076 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:44.030085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:44.030326 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:44.030523 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:44.031126 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.032445 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.034922 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.036681 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.036803 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.037015 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:44.037557 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 13:00:44.037716 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":458,"threads_started":1}
I20260619 13:00:44.041010 5766 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec123e056914393a1963cb352253920 (DEFAULT_TABLE table=test-table [id=bb2f5af703d940e595243e41de05bc94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:44.042301 5766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec123e056914393a1963cb352253920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:44.042038 5633 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec123e056914393a1963cb352253920 (DEFAULT_TABLE table=test-table [id=bb2f5af703d940e595243e41de05bc94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:44.043356 5633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec123e056914393a1963cb352253920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:44.043143 5898 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec123e056914393a1963cb352253920 (DEFAULT_TABLE table=test-table [id=bb2f5af703d940e595243e41de05bc94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:44.044445 5898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec123e056914393a1963cb352253920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:44.054391 5981 tablet_bootstrap.cc:492] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: Bootstrap starting.
I20260619 13:00:44.057170 5982 tablet_bootstrap.cc:492] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: Bootstrap starting.
I20260619 13:00:44.058023 5981 tablet_bootstrap.cc:654] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:44.059238 5981 log.cc:824] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:44.060202 5982 tablet_bootstrap.cc:654] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:44.061107 5984 tablet_bootstrap.cc:492] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: Bootstrap starting.
I20260619 13:00:44.061410 5982 log.cc:824] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:44.064347 5981 tablet_bootstrap.cc:492] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: No bootstrap required, opened a new log
I20260619 13:00:44.064615 5981 ts_tablet_manager.cc:1397] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 13:00:44.065276 5984 tablet_bootstrap.cc:654] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:44.066718 5984 log.cc:824] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:44.069787 5984 tablet_bootstrap.cc:492] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: No bootstrap required, opened a new log
I20260619 13:00:44.070048 5984 ts_tablet_manager.cc:1397] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 13:00:44.070834 5982 tablet_bootstrap.cc:492] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: No bootstrap required, opened a new log
I20260619 13:00:44.071080 5982 ts_tablet_manager.cc:1397] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260619 13:00:44.072871 5981 raft_consensus.cc:348] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } }
I20260619 13:00:44.073223 5981 raft_consensus.cc:374] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:44.073310 5981 raft_consensus.cc:729] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33ffd20763b4d5d8fbc793e862a2023, State: Initialized, Role: FOLLOWER
I20260619 13:00:44.073874 5981 consensus_queue.cc:260] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } }
I20260619 13:00:44.074923 5831 heartbeater.cc:503] Master 127.1.87.126:36995 was elected leader, sending a full tablet report...
I20260619 13:00:44.075460 5981 ts_tablet_manager.cc:1428] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260619 13:00:44.079923 5984 raft_consensus.cc:348] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } }
I20260619 13:00:44.080322 5984 raft_consensus.cc:374] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:44.080422 5984 raft_consensus.cc:729] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff7556c661a47cfb3613fc21a331924, State: Initialized, Role: FOLLOWER
I20260619 13:00:44.080914 5982 raft_consensus.cc:348] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } }
I20260619 13:00:44.081051 5984 consensus_queue.cc:260] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } }
I20260619 13:00:44.081277 5982 raft_consensus.cc:374] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:44.081367 5982 raft_consensus.cc:729] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a62c7ec8374ddea7ce178377f278e5, State: Initialized, Role: FOLLOWER
I20260619 13:00:44.081884 5982 consensus_queue.cc:260] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } }
I20260619 13:00:44.082412 5963 heartbeater.cc:503] Master 127.1.87.126:36995 was elected leader, sending a full tablet report...
I20260619 13:00:44.082805 5984 ts_tablet_manager.cc:1428] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 13:00:44.084585 5982 ts_tablet_manager.cc:1428] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260619 13:00:44.188601 5988 raft_consensus.cc:482] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:44.188912 5988 raft_consensus.cc:504] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } }
I20260619 13:00:44.190631 5988 leader_election.cc:288] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4a62c7ec8374ddea7ce178377f278e5 (127.1.87.65:41759), d33ffd20763b4d5d8fbc793e862a2023 (127.1.87.66:43289)
W20260619 13:00:44.191133 5964 tablet.cc:2369] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:00:44.197484 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.191106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:43289 (local address 127.1.87.67:59137)
0619 13:00:44.191469 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:44.191491 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:44.191639 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:44.192107 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:44.192114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:44.192148 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:44.192366 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.192381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.193623 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.193627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.195526 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.195540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.196894 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.196898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.197032 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.197063 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:00:44.197134 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 13:00:44.197240 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260619 13:00:44.197566 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.191209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:59137 (local address 127.1.87.66:43289)
0619 13:00:44.191401 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:44.191408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:44.191587 (+ 179us) server_negotiation.cc:407] Connection header received
0619 13:00:44.191798 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:44.191806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:44.191904 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:44.192018 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:44.192514 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.193502 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.195682 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.196743 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.196886 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.196982 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:44.197182 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 13:00:44.197310 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:00:44.198262 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.191122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:51957 (local address 127.1.87.65:41759)
0619 13:00:44.191363 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:44.191370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:44.192414 (+ 1044us) server_negotiation.cc:407] Connection header received
0619 13:00:44.192559 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:44.192566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:44.192637 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:44.192732 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:44.193163 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.194013 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.196147 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.197296 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.197427 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.197518 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:44.197896 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 13:00:44.198015 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:00:44.198288 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.191532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:41759 (local address 127.1.87.67:51957)
0619 13:00:44.192305 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:44.192323 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:44.192437 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:44.192747 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:44.192750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:44.192768 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:44.193027 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.193033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.194153 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.194158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.195985 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:44.195994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.197513 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.197520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:44.197727 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.197765 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:00:44.197855 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 13:00:44.197968 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":431,"threads_started":1}
I20260619 13:00:44.198419 5786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec123e056914393a1963cb352253920" candidate_uuid: "aff7556c661a47cfb3613fc21a331924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33ffd20763b4d5d8fbc793e862a2023" is_pre_election: true
I20260619 13:00:44.198984 5786 raft_consensus.cc:2454] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff7556c661a47cfb3613fc21a331924 in term 0.
I20260619 13:00:44.199429 5653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec123e056914393a1963cb352253920" candidate_uuid: "aff7556c661a47cfb3613fc21a331924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4a62c7ec8374ddea7ce178377f278e5" is_pre_election: true
I20260619 13:00:44.199755 5851 leader_election.cc:302] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aff7556c661a47cfb3613fc21a331924, d33ffd20763b4d5d8fbc793e862a2023; no voters:
I20260619 13:00:44.199862 5653 raft_consensus.cc:2454] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff7556c661a47cfb3613fc21a331924 in term 0.
I20260619 13:00:44.200121 5988 raft_consensus.cc:2781] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:44.200269 5988 raft_consensus.cc:482] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:44.200389 5988 raft_consensus.cc:3037] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:44.202433 5988 raft_consensus.cc:504] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } }
I20260619 13:00:44.203017 5988 leader_election.cc:288] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [CANDIDATE]: Term 1 election: Requested vote from peers f4a62c7ec8374ddea7ce178377f278e5 (127.1.87.65:41759), d33ffd20763b4d5d8fbc793e862a2023 (127.1.87.66:43289)
I20260619 13:00:44.203424 5653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec123e056914393a1963cb352253920" candidate_uuid: "aff7556c661a47cfb3613fc21a331924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4a62c7ec8374ddea7ce178377f278e5"
I20260619 13:00:44.203454 5786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec123e056914393a1963cb352253920" candidate_uuid: "aff7556c661a47cfb3613fc21a331924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33ffd20763b4d5d8fbc793e862a2023"
I20260619 13:00:44.203601 5653 raft_consensus.cc:3037] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:44.203625 5786 raft_consensus.cc:3037] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:44.205749 5653 raft_consensus.cc:2454] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff7556c661a47cfb3613fc21a331924 in term 1.
I20260619 13:00:44.205888 5786 raft_consensus.cc:2454] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff7556c661a47cfb3613fc21a331924 in term 1.
I20260619 13:00:44.206130 5852 leader_election.cc:302] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aff7556c661a47cfb3613fc21a331924, f4a62c7ec8374ddea7ce178377f278e5; no voters:
I20260619 13:00:44.206400 5988 raft_consensus.cc:2781] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:44.206825 5988 raft_consensus.cc:686] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 1 LEADER]: Becoming Leader. State: Replica: aff7556c661a47cfb3613fc21a331924, State: Running, Role: LEADER
I20260619 13:00:44.207271 5988 consensus_queue.cc:237] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } }
I20260619 13:00:44.211750 5510 catalog_manager.cc:5495] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 reported cstate change: term changed from 0 to 1, leader changed from <none> to aff7556c661a47cfb3613fc21a331924 (127.1.87.67). New cstate: current_term: 1 leader_uuid: "aff7556c661a47cfb3613fc21a331924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aff7556c661a47cfb3613fc21a331924" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39969 } health_report { overall_health: HEALTHY } } }
W20260619 13:00:44.213951 5832 tablet.cc:2369] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:00:44.232265 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.227389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.1.87.67:39969)
0619 13:00:44.227556 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:44.227560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:44.227719 (+ 159us) server_negotiation.cc:407] Connection header received
0619 13:00:44.227897 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:44.227901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:44.227972 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:44.228099 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:00:44.228485 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.229179 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.230425 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.230809 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.230906 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.231295 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:00:44.231843 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:00:44.231961 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:44.232022 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 13:00:44.232081 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:00:44.240173 5786 raft_consensus.cc:1261] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [term 1 FOLLOWER]: Refusing update from remote peer aff7556c661a47cfb3613fc21a331924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:44.240186 5653 raft_consensus.cc:1261] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [term 1 FOLLOWER]: Refusing update from remote peer aff7556c661a47cfb3613fc21a331924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:44.241117 5991 consensus_queue.cc:1035] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33ffd20763b4d5d8fbc793e862a2023" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:44.241453 5988 consensus_queue.cc:1035] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4a62c7ec8374ddea7ce178377f278e5" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:00:44.242702 5699 tablet.cc:2369] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:00:44.251781 5994 mvcc.cc:205] Tried to move back new op lower bound from 7298556085197516800 to 7298556085076541440. Current Snapshot: MvccSnapshot[applied={T|T < 7298556085197516800}]
I20260619 13:00:44.252398 5996 mvcc.cc:205] Tried to move back new op lower bound from 7298556085197516800 to 7298556085076541440. Current Snapshot: MvccSnapshot[applied={T|T < 7298556085197516800}]
I20260619 13:00:44.252777 5995 mvcc.cc:205] Tried to move back new op lower bound from 7298556085197516800 to 7298556085076541440. Current Snapshot: MvccSnapshot[applied={T|T < 7298556085197516800}]
I20260619 13:00:44.269162 5509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:44.269428 5509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:44.272411 5509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e15bd846fd354741971125dbec4e14c4: Sending DeleteTablet for 3 replicas of tablet 1ec123e056914393a1963cb352253920
I20260619 13:00:44.273461 5766 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec123e056914393a1963cb352253920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:44 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:42742
I20260619 13:00:44.273659 5633 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec123e056914393a1963cb352253920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:44 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:51434
I20260619 13:00:44.274307 5898 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec123e056914393a1963cb352253920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:44 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:35590
I20260619 13:00:44.274500 6005 tablet_replica.cc:330] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: stopping tablet replica
I20260619 13:00:44.274897 6005 raft_consensus.cc:2229] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:44.275419 6005 raft_consensus.cc:2258] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:44.276289 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 5571
I20260619 13:00:44.277559 6005 ts_tablet_manager.cc:1905] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:44.285581 6005 ts_tablet_manager.cc:1918] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:00:44.285832 6005 log.cc:1196] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/1ec123e056914393a1963cb352253920
I20260619 13:00:44.286506 6005 ts_tablet_manager.cc:1939] T 1ec123e056914393a1963cb352253920 P f4a62c7ec8374ddea7ce178377f278e5: Deleting consensus metadata
I20260619 13:00:44.288542 5495 catalog_manager.cc:4841] TS f4a62c7ec8374ddea7ce178377f278e5 (127.1.87.65:41759): tablet 1ec123e056914393a1963cb352253920 (table test-table [id=bb2f5af703d940e595243e41de05bc94]) successfully deleted
I20260619 13:00:44.292483 6006 tablet_replica.cc:330] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: stopping tablet replica
I20260619 13:00:44.292985 6006 raft_consensus.cc:2229] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:00:44.293800 6006 raft_consensus.cc:2258] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:44.294694 6004 tablet_replica.cc:330] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: stopping tablet replica
I20260619 13:00:44.295742 6004 raft_consensus.cc:2229] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:44.296231 6004 raft_consensus.cc:2258] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:44.296475 6006 ts_tablet_manager.cc:1905] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:44.298161 6004 ts_tablet_manager.cc:1905] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:00:44 dist-test-slave-tpsm krb5kdc[5466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:44.304592 6006 ts_tablet_manager.cc:1918] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:00:44.304850 6006 log.cc:1196] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/1ec123e056914393a1963cb352253920
I20260619 13:00:44.304895 6004 ts_tablet_manager.cc:1918] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:00:44.305052 6004 log.cc:1196] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/1ec123e056914393a1963cb352253920
I20260619 13:00:44.305595 6004 ts_tablet_manager.cc:1939] T 1ec123e056914393a1963cb352253920 P d33ffd20763b4d5d8fbc793e862a2023: Deleting consensus metadata
I20260619 13:00:44.305868 6006 ts_tablet_manager.cc:1939] T 1ec123e056914393a1963cb352253920 P aff7556c661a47cfb3613fc21a331924: Deleting consensus metadata
I20260619 13:00:44.307422 5495 catalog_manager.cc:4841] TS aff7556c661a47cfb3613fc21a331924 (127.1.87.67:39969): tablet 1ec123e056914393a1963cb352253920 (table test-table [id=bb2f5af703d940e595243e41de05bc94]) successfully deleted
I20260619 13:00:44.307864 5494 catalog_manager.cc:4841] TS d33ffd20763b4d5d8fbc793e862a2023 (127.1.87.66:43289): tablet 1ec123e056914393a1963cb352253920 (table test-table [id=bb2f5af703d940e595243e41de05bc94]) successfully deleted
I20260619 13:00:44.309998 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.278012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.1.87.65:41759)
0619 13:00:44.278195 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:44.278202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:44.291830 (+ 13628us) server_negotiation.cc:407] Connection header received
0619 13:00:44.292029 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:44.292036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:44.292130 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:44.292227 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:44.294028 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.295141 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.297013 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.297627 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.297743 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.303222 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:44.303240 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:44.303257 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:44.303305 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:44.307226 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:44.308117 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:44.308124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:44.308126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:44.308264 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:44.308651 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:44.308654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:44.308656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:44.308985 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:44.309188 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:44.309552 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 13:00:44.309705 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:00:44.310700 5693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:44.399974 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 5571
I20260619 13:00:44.417318 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 5704
Jun 19 13:00:44 dist-test-slave-tpsm krb5kdc[5466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:00:44.432915 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.418438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.1.87.66:43289)
0619 13:00:44.418629 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:44.418642 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:44.418658 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:00:44.418708 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:44.418711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:44.418802 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:44.418920 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:44.420237 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.420967 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.422185 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.422590 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.422726 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.426919 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:44.426934 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:44.426951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:44.426989 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:44.429832 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:44.430515 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:44.430522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:44.430525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:44.430649 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:44.431070 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:44.431075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:44.431080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:44.431703 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:44.432015 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:44.432333 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 13:00:44.432567 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 13:00:44.433559 5826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:44.529140 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 5704
I20260619 13:00:44.552068 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 5836
Jun 19 13:00:44 dist-test-slave-tpsm krb5kdc[5466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:00:44.568002 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:44.552836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.1.87.67:39969)
0619 13:00:44.553036 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:44.553042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:44.553062 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:00:44.553204 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:44.553209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:44.553277 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:44.553396 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:44.554632 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.555521 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.556709 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:44.557168 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:44.557326 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:44.562085 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:44.562101 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:44.562116 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:44.562160 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:44.565240 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:44.565893 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:44.565901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:44.565903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:44.566056 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:44.566457 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:44.566463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:44.566466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:44.566802 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:44.567121 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:44.567417 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 13:00:44.567626 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:00:44.568625 5958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:44.686687 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 5836
I20260619 13:00:44.705225 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 5479
I20260619 13:00:44.706285 5540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:44.812055 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 5479
2026-06-19T13:00:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4117 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T13:00:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:44Z Disabled control of system clock
I20260619 13:00:44.876152 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39193
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:39193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 13:00:45.192476 6015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:45.192785 6015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:45.192938 6015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:45.200328 6015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:45.200435 6015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:45.200536 6015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:45.200654 6015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:45.200728 6015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:45.210728 6015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:39193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:45.212052 6015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:45.213863 6015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:45.225687 6021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:45.225823 6020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:45.225824 6023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:45.226538 6015 server_base.cc:1034] running on GCE node
I20260619 13:00:45.227444 6015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:45.228768 6015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:45.230000 6015 hybrid_clock.cc:648] HybridClock initialized: now 1781874045229959 us; error 59 us; skew 500 ppm
I20260619 13:00:45.230370 6015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:00:45.236646 6015 webserver.cc:460] Webserver started at http://127.1.87.126:36341/ using document root <none> and password file <none>
I20260619 13:00:45.237357 6015 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:45.237490 6015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:45.237885 6015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:45.240846 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "42420bcaddbf458da39ad5bfa8ab309b"
format_stamp: "Formatted at 2026-06-19 13:00:45 on dist-test-slave-tpsm"
server_key: "5a22ba41a931cff3eb62df039ffefffc"
server_key_iv: "7222622e82cfd1a2d20aebc11af8942b"
server_key_version: "encryptionkey@0"
I20260619 13:00:45.241639 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "42420bcaddbf458da39ad5bfa8ab309b"
format_stamp: "Formatted at 2026-06-19 13:00:45 on dist-test-slave-tpsm"
server_key: "5a22ba41a931cff3eb62df039ffefffc"
server_key_iv: "7222622e82cfd1a2d20aebc11af8942b"
server_key_version: "encryptionkey@0"
I20260619 13:00:45.246696 6015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:00:45.249977 6029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:45.251174 6015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:45.251406 6015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "42420bcaddbf458da39ad5bfa8ab309b"
format_stamp: "Formatted at 2026-06-19 13:00:45 on dist-test-slave-tpsm"
server_key: "5a22ba41a931cff3eb62df039ffefffc"
server_key_iv: "7222622e82cfd1a2d20aebc11af8942b"
server_key_version: "encryptionkey@0"
I20260619 13:00:45.251623 6015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:45.257071 6015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:45.257789 6015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:45.258074 6015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:45.275897 6015 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:39193
I20260619 13:00:45.275938 6080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:39193 every 8 connection(s)
I20260619 13:00:45.277204 6015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:45.280755 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:45.283942 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 6015
I20260619 13:00:45.284138 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:45.284919 1373 external_mini_cluster.cc:1428] Setting key 7008906b831be5d9c148f529b5d4d5d6
I20260619 13:00:45.293108 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:45.287069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.1.87.126:39193)
0619 13:00:45.288082 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:45.288092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:45.288130 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:00:45.289613 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:45.289635 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:45.290056 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:45.290442 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:45.291532 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:45.291537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:45.291549 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:45.291573 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:45.291627 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:45.291642 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:45.291881 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:45.292018 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 13:00:45.292259 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":426,"threads_started":1}
I20260619 13:00:45.293220 6081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b: Bootstrap starting.
I20260619 13:00:45.296345 6081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:45.297317 6081 log.cc:824] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:45.300144 6081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b: No bootstrap required, opened a new log
I20260619 13:00:45.307425 6081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42420bcaddbf458da39ad5bfa8ab309b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39193 } }
I20260619 13:00:45.307893 6081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:45.307979 6081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42420bcaddbf458da39ad5bfa8ab309b, State: Initialized, Role: FOLLOWER
I20260619 13:00:45.308497 6081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42420bcaddbf458da39ad5bfa8ab309b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39193 } }
I20260619 13:00:45.308699 6081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:45.308790 6081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:45.308900 6081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:45.311269 6081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42420bcaddbf458da39ad5bfa8ab309b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39193 } }
I20260619 13:00:45.311767 6081 leader_election.cc:302] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42420bcaddbf458da39ad5bfa8ab309b; no voters:
I20260619 13:00:45.312258 6081 leader_election.cc:288] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:45.312397 6086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:45.313090 6086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [term 1 LEADER]: Becoming Leader. State: Replica: 42420bcaddbf458da39ad5bfa8ab309b, State: Running, Role: LEADER
I20260619 13:00:45.313625 6086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42420bcaddbf458da39ad5bfa8ab309b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39193 } }
I20260619 13:00:45.314306 6081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:45.316425 6087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42420bcaddbf458da39ad5bfa8ab309b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42420bcaddbf458da39ad5bfa8ab309b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39193 } } }
I20260619 13:00:45.316457 6088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42420bcaddbf458da39ad5bfa8ab309b. Latest consensus state: current_term: 1 leader_uuid: "42420bcaddbf458da39ad5bfa8ab309b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42420bcaddbf458da39ad5bfa8ab309b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39193 } } }
I20260619 13:00:45.316781 6087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:45.316867 6088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:45.318727 6091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:45.324771 6091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:45.333174 6091 catalog_manager.cc:1347] Generated new cluster ID: e9c55e816bdf4f6f832cfea9e411c183
I20260619 13:00:45.333282 6091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:45.345137 6091 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:45.345558 6091 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:45.353375 6091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b: Generated new TSK 0
I20260619 13:00:45.354100 6091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:00:45.363721 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39193
--builtin_ntp_servers=127.1.87.84:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 13:00:45.694425 6105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:45.694685 6105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:45.694774 6105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:45.701918 6105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:45.702019 6105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:45.702128 6105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:45.711220 6105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:45.712459 6105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:45.714140 6105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:45.727028 6110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:45.727099 6113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:45.727062 6111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:45.728022 6105 server_base.cc:1034] running on GCE node
I20260619 13:00:45.728516 6105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:45.729359 6105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:45.730556 6105 hybrid_clock.cc:648] HybridClock initialized: now 1781874045730517 us; error 28 us; skew 500 ppm
I20260619 13:00:45.730824 6105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:00:45.733009 6105 webserver.cc:460] Webserver started at http://127.1.87.65:41885/ using document root <none> and password file <none>
I20260619 13:00:45.733630 6105 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:45.733726 6105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:45.733978 6105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:45.736663 6105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "2404cfdf892b4c67b8ff8c59a0c796fd"
format_stamp: "Formatted at 2026-06-19 13:00:45 on dist-test-slave-tpsm"
server_key: "5bc36c071d8e5d130c9c456b08f27494"
server_key_iv: "0c7b1de7d1836fd2fbf3634bc62385d0"
server_key_version: "encryptionkey@0"
I20260619 13:00:45.737366 6105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "2404cfdf892b4c67b8ff8c59a0c796fd"
format_stamp: "Formatted at 2026-06-19 13:00:45 on dist-test-slave-tpsm"
server_key: "5bc36c071d8e5d130c9c456b08f27494"
server_key_iv: "0c7b1de7d1836fd2fbf3634bc62385d0"
server_key_version: "encryptionkey@0"
I20260619 13:00:45.743109 6105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:00:45.746528 6119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:45.747929 6105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:45.748106 6105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "2404cfdf892b4c67b8ff8c59a0c796fd"
format_stamp: "Formatted at 2026-06-19 13:00:45 on dist-test-slave-tpsm"
server_key: "5bc36c071d8e5d130c9c456b08f27494"
server_key_iv: "0c7b1de7d1836fd2fbf3634bc62385d0"
server_key_version: "encryptionkey@0"
I20260619 13:00:45.748351 6105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:45.754043 6105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:45.754912 6105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:45.755193 6105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:45.756188 6105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:45.757581 6105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:45.757699 6105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:45.757836 6105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:45.757923 6105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:45.790474 6105 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:40985
I20260619 13:00:45.790530 6231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:40985 every 8 connection(s)
I20260619 13:00:45.791771 6105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:45.794293 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 6105
I20260619 13:00:45.794512 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:45.795360 1373 external_mini_cluster.cc:1428] Setting key 71e9462d37a4773926b66f4122d85ebe
I20260619 13:00:45.798318 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39193
--builtin_ntp_servers=127.1.87.84:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 13:00:45.800568 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:45.794541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:58517 (local address 127.1.87.126:39193)
0619 13:00:45.795040 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:45.795047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:45.796288 (+ 1241us) server_negotiation.cc:407] Connection header received
0619 13:00:45.798086 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:45.798093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:45.798158 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:45.798287 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:45.799748 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:45.799757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:45.799765 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:45.799788 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:45.799841 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:45.799859 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:45.799984 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:45.800142 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 13:00:45.800209 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":227,"threads_started":1}
I20260619 13:00:45.801235 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:45.795076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39193 (local address 127.1.87.65:58517)
0619 13:00:45.796124 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:45.796170 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:45.797835 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:45.798516 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:45.798531 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:45.798666 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:45.799453 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:00:45.799455 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:45.799490 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:00:45.799570 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:45.799960 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:45.799967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:45.799971 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 13:00:45.800170 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 13:00:45.800348 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":375,"threads_started":1}
I20260619 13:00:45.802574 6232 heartbeater.cc:347] Connected to a master server at 127.1.87.126:39193
I20260619 13:00:45.802875 6232 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:45.803475 6232 heartbeater.cc:511] Master 127.1.87.126:39193 requested a full tablet report, sending...
I20260619 13:00:45.805334 6046 ts_manager.cc:194] Registered new tserver with Master: 2404cfdf892b4c67b8ff8c59a0c796fd (127.1.87.65:40985)
W20260619 13:00:46.132172 6236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:46.132483 6236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:46.132642 6236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:46.139943 6236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:46.140079 6236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:46.140254 6236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:46.149507 6236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:46.150816 6236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:46.152751 6236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:46.166504 6242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:46.166594 6241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:46.166548 6244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:46.167902 6236 server_base.cc:1034] running on GCE node
I20260619 13:00:46.168529 6236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:46.169569 6236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:46.170801 6236 hybrid_clock.cc:648] HybridClock initialized: now 1781874046170753 us; error 48 us; skew 500 ppm
I20260619 13:00:46.171123 6236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:00:46.173538 6236 webserver.cc:460] Webserver started at http://127.1.87.66:34649/ using document root <none> and password file <none>
I20260619 13:00:46.174283 6236 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:46.174420 6236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:46.174737 6236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:46.177652 6236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "28eccbfd381040719fe51f2551bc4083"
format_stamp: "Formatted at 2026-06-19 13:00:46 on dist-test-slave-tpsm"
server_key: "5e711deafe6417836e1093857ad45606"
server_key_iv: "8fd668f0d7080f0a4c383742f398b5b1"
server_key_version: "encryptionkey@0"
I20260619 13:00:46.178465 6236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "28eccbfd381040719fe51f2551bc4083"
format_stamp: "Formatted at 2026-06-19 13:00:46 on dist-test-slave-tpsm"
server_key: "5e711deafe6417836e1093857ad45606"
server_key_iv: "8fd668f0d7080f0a4c383742f398b5b1"
server_key_version: "encryptionkey@0"
I20260619 13:00:46.183801 6236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:00:46.187191 6250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:46.188757 6236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:46.188939 6236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "28eccbfd381040719fe51f2551bc4083"
format_stamp: "Formatted at 2026-06-19 13:00:46 on dist-test-slave-tpsm"
server_key: "5e711deafe6417836e1093857ad45606"
server_key_iv: "8fd668f0d7080f0a4c383742f398b5b1"
server_key_version: "encryptionkey@0"
I20260619 13:00:46.189196 6236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:46.194696 6236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:46.195513 6236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:46.195786 6236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:46.196668 6236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:46.198015 6236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:46.198140 6236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:46.198282 6236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:46.198374 6236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:46.230822 6236 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:35807
I20260619 13:00:46.230913 6362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:35807 every 8 connection(s)
I20260619 13:00:46.232120 6236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:46.237464 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 6236
I20260619 13:00:46.237610 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:46.238049 1373 external_mini_cluster.cc:1428] Setting key 745b37c0d44e3da9443ab9af50fe7c2c
I20260619 13:00:46.240589 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.234934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:58495 (local address 127.1.87.126:39193)
0619 13:00:46.235108 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.235113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.236502 (+ 1389us) server_negotiation.cc:407] Connection header received
0619 13:00:46.238321 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.238327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.238394 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.238510 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.239877 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.239883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.239888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.239905 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.239946 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.239957 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.240067 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.240265 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.240326 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:00:46.240796 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39193
--builtin_ntp_servers=127.1.87.84:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 13:00:46.241595 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.235501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39193 (local address 127.1.87.66:58495)
0619 13:00:46.236352 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:46.236390 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:46.238039 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:46.238733 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:46.238740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:46.238820 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:46.239559 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:00:46.239562 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:46.239605 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:00:46.239717 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:46.240055 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:46.240064 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:46.240072 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 13:00:46.240343 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 13:00:46.240570 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":270,"threads_started":1}
I20260619 13:00:46.244253 6363 heartbeater.cc:347] Connected to a master server at 127.1.87.126:39193
I20260619 13:00:46.244539 6363 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:46.245074 6363 heartbeater.cc:511] Master 127.1.87.126:39193 requested a full tablet report, sending...
I20260619 13:00:46.246336 6046 ts_manager.cc:194] Registered new tserver with Master: 28eccbfd381040719fe51f2551bc4083 (127.1.87.66:35807)
W20260619 13:00:46.559337 6366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:46.559628 6366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:46.559780 6366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:46.566932 6366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:46.567037 6366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:46.567199 6366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:46.577296 6366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:46.578660 6366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:46.580394 6366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:46.593045 6372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:46.593067 6374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:46.593169 6371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:46.593931 6366 server_base.cc:1034] running on GCE node
I20260619 13:00:46.594529 6366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:46.595465 6366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:46.596664 6366 hybrid_clock.cc:648] HybridClock initialized: now 1781874046596614 us; error 43 us; skew 500 ppm
I20260619 13:00:46.596930 6366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:00:46.599088 6366 webserver.cc:460] Webserver started at http://127.1.87.67:46505/ using document root <none> and password file <none>
I20260619 13:00:46.599767 6366 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:46.599851 6366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:46.600090 6366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:46.602705 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "b317e29a30a44aa7aa0288baf0f40117"
format_stamp: "Formatted at 2026-06-19 13:00:46 on dist-test-slave-tpsm"
server_key: "6b054f3520ccac0766301955f7c97634"
server_key_iv: "a26fff628f4ea1c1402d81dcb750fbda"
server_key_version: "encryptionkey@0"
I20260619 13:00:46.603466 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "b317e29a30a44aa7aa0288baf0f40117"
format_stamp: "Formatted at 2026-06-19 13:00:46 on dist-test-slave-tpsm"
server_key: "6b054f3520ccac0766301955f7c97634"
server_key_iv: "a26fff628f4ea1c1402d81dcb750fbda"
server_key_version: "encryptionkey@0"
I20260619 13:00:46.608206 6366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:00:46.611552 6380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:46.612777 6366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:00:46.612953 6366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "b317e29a30a44aa7aa0288baf0f40117"
format_stamp: "Formatted at 2026-06-19 13:00:46 on dist-test-slave-tpsm"
server_key: "6b054f3520ccac0766301955f7c97634"
server_key_iv: "a26fff628f4ea1c1402d81dcb750fbda"
server_key_version: "encryptionkey@0"
I20260619 13:00:46.613181 6366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:46.618618 6366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:46.619462 6366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:46.619730 6366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:46.620713 6366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:46.622104 6366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:46.622223 6366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:46.622359 6366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:46.622445 6366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:46.654819 6366 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:41865
I20260619 13:00:46.654839 6492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:41865 every 8 connection(s)
I20260619 13:00:46.656205 6366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:46.659018 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 6366
I20260619 13:00:46.659176 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:46.659648 1373 external_mini_cluster.cc:1428] Setting key 412f651f0ae6862d4c1a337fdde35c1e
I20260619 13:00:46.664212 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.658568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:60707 (local address 127.1.87.126:39193)
0619 13:00:46.658828 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.658836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.660361 (+ 1525us) server_negotiation.cc:407] Connection header received
0619 13:00:46.661990 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.661995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.662058 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.662152 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.663474 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.663480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.663485 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.663503 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.663543 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.663553 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.663675 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.663869 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.663931 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 13:00:46.665019 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.659052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39193 (local address 127.1.87.67:60707)
0619 13:00:46.660211 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:46.660254 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:46.661746 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:46.662382 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:46.662390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:46.662476 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:46.663158 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:00:46.663161 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:46.663197 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:00:46.663273 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:46.663657 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:46.663667 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:46.663683 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 13:00:46.663946 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 13:00:46.664169 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":425,"threads_started":1}
I20260619 13:00:46.666168 6493 heartbeater.cc:347] Connected to a master server at 127.1.87.126:39193
I20260619 13:00:46.666442 6493 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:46.666977 6493 heartbeater.cc:511] Master 127.1.87.126:39193 requested a full tablet report, sending...
I20260619 13:00:46.668286 6046 ts_manager.cc:194] Registered new tserver with Master: b317e29a30a44aa7aa0288baf0f40117 (127.1.87.67:41865)
I20260619 13:00:46.674278 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:00:46.680790 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.678602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.1.87.126:39193)
0619 13:00:46.678759 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.678763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.679252 (+ 489us) server_negotiation.cc:407] Connection header received
0619 13:00:46.679428 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.679432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.679499 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.679611 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.680379 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.680383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.680386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.680400 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.680428 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.680439 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.680528 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.680559 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.680598 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:00:46.684244 6046 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:00:46.690903 6046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:00:46.717548 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.703920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.1.87.67:41865)
0619 13:00:46.713332 (+ 9412us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.713343 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.714574 (+ 1231us) server_negotiation.cc:407] Connection header received
0619 13:00:46.714777 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.714784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.714976 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.715109 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.716447 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.716455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.716469 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.716485 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.716530 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.716543 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.716709 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.717042 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.717143 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9199,"thread_start_us":9010,"threads_started":1}
I20260619 13:00:46.717589 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.702048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40985 (local address 127.0.0.1:50112)
0619 13:00:46.702798 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:46.702831 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:46.702949 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:46.714627 (+ 11678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:46.714636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:46.714740 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:46.715708 (+ 968us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:00:46.715713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:46.715752 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:00:46.715833 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:46.716896 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:46.716901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:46.716905 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 13:00:46.717050 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 13:00:46.717091 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":336,"threads_started":1}
I20260619 13:00:46.718125 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.702656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:35807 (local address 127.0.0.1:57768)
0619 13:00:46.713702 (+ 11046us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:46.713740 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:46.713898 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:46.714627 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:46.714636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:46.714740 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:46.716079 (+ 1339us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:00:46.716084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:46.716103 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:00:46.716149 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:46.717486 (+ 1337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:46.717491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:46.717495 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 13:00:46.717628 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 13:00:46.717676 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10825,"thread_start_us":10708,"threads_started":1}
I20260619 13:00:46.718429 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.703121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:41865 (local address 127.0.0.1:60980)
0619 13:00:46.714402 (+ 11281us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:46.714420 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:46.714510 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:46.715299 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:46.715303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:46.715353 (+ 50us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:46.716020 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:00:46.716023 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:46.716037 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:00:46.716076 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:46.716760 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:46.716767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:46.716773 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:00:46.716959 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 13:00:46.717053 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11146,"thread_start_us":11033,"threads_started":1}
I20260619 13:00:46.718688 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.702255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.1.87.65:40985)
0619 13:00:46.714058 (+ 11803us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.714068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.714090 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:00:46.714183 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.714189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.714393 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.715264 (+ 871us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.716662 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.716672 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.716688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.716708 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.716763 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.716786 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.716927 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.718261 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.718357 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11654,"thread_start_us":11534,"threads_started":1}
I20260619 13:00:46.719599 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.703259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.1.87.66:35807)
0619 13:00:46.712709 (+ 9450us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.712722 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.714043 (+ 1321us) server_negotiation.cc:407] Connection header received
0619 13:00:46.714133 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.714142 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.714394 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.716315 (+ 1921us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.717245 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.717253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.717264 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.717281 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.717332 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.717349 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.718411 (+ 1062us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.718479 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.718579 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9208,"thread_start_us":9054,"threads_started":1}
I20260619 13:00:46.722379 6167 tablet_service.cc:1467] Processing CreateTablet for tablet 1c659f9a6bd447889e39b5969dd52155 (DEFAULT_TABLE table=test-table [id=4dae901f1bfe438d80d3dcae60476ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:46.722385 6428 tablet_service.cc:1467] Processing CreateTablet for tablet 1c659f9a6bd447889e39b5969dd52155 (DEFAULT_TABLE table=test-table [id=4dae901f1bfe438d80d3dcae60476ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:46.722810 6298 tablet_service.cc:1467] Processing CreateTablet for tablet 1c659f9a6bd447889e39b5969dd52155 (DEFAULT_TABLE table=test-table [id=4dae901f1bfe438d80d3dcae60476ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:00:46.723690 6167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c659f9a6bd447889e39b5969dd52155. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:46.723759 6428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c659f9a6bd447889e39b5969dd52155. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:46.723958 6298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c659f9a6bd447889e39b5969dd52155. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:46.735124 6511 tablet_bootstrap.cc:492] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd: Bootstrap starting.
I20260619 13:00:46.738004 6513 tablet_bootstrap.cc:492] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: Bootstrap starting.
I20260619 13:00:46.739152 6511 tablet_bootstrap.cc:654] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:46.740551 6511 log.cc:824] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:46.741696 6513 tablet_bootstrap.cc:654] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:46.741961 6512 tablet_bootstrap.cc:492] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: Bootstrap starting.
I20260619 13:00:46.742883 6513 log.cc:824] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:46.744529 6511 tablet_bootstrap.cc:492] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd: No bootstrap required, opened a new log
I20260619 13:00:46.744820 6511 ts_tablet_manager.cc:1397] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 13:00:46.745256 6512 tablet_bootstrap.cc:654] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:46.746137 6512 log.cc:824] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:46.746253 6513 tablet_bootstrap.cc:492] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: No bootstrap required, opened a new log
I20260619 13:00:46.746490 6513 ts_tablet_manager.cc:1397] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260619 13:00:46.748505 6512 tablet_bootstrap.cc:492] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: No bootstrap required, opened a new log
I20260619 13:00:46.748749 6512 ts_tablet_manager.cc:1397] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 13:00:46.755266 6511 raft_consensus.cc:348] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } }
I20260619 13:00:46.755759 6511 raft_consensus.cc:374] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:46.755865 6511 raft_consensus.cc:729] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2404cfdf892b4c67b8ff8c59a0c796fd, State: Initialized, Role: FOLLOWER
I20260619 13:00:46.756248 6512 raft_consensus.cc:348] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } }
I20260619 13:00:46.756605 6512 raft_consensus.cc:374] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:46.756461 6511 consensus_queue.cc:260] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } }
I20260619 13:00:46.756726 6512 raft_consensus.cc:729] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b317e29a30a44aa7aa0288baf0f40117, State: Initialized, Role: FOLLOWER
I20260619 13:00:46.756711 6513 raft_consensus.cc:348] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } }
I20260619 13:00:46.757149 6513 raft_consensus.cc:374] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:46.757252 6513 raft_consensus.cc:729] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eccbfd381040719fe51f2551bc4083, State: Initialized, Role: FOLLOWER
I20260619 13:00:46.757421 6512 consensus_queue.cc:260] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } }
I20260619 13:00:46.757725 6232 heartbeater.cc:503] Master 127.1.87.126:39193 was elected leader, sending a full tablet report...
I20260619 13:00:46.757766 6513 consensus_queue.cc:260] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } }
I20260619 13:00:46.758510 6511 ts_tablet_manager.cc:1428] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260619 13:00:46.760175 6493 heartbeater.cc:503] Master 127.1.87.126:39193 was elected leader, sending a full tablet report...
I20260619 13:00:46.760241 6512 ts_tablet_manager.cc:1428] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 13:00:46.760741 6513 ts_tablet_manager.cc:1428] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260619 13:00:46.760769 6363 heartbeater.cc:503] Master 127.1.87.126:39193 was elected leader, sending a full tablet report...
W20260619 13:00:46.794039 6233 tablet.cc:2369] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:00:46.820896 6518 raft_consensus.cc:482] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:46.821169 6518 raft_consensus.cc:504] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } }
I20260619 13:00:46.822703 6518 leader_election.cc:288] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2404cfdf892b4c67b8ff8c59a0c796fd (127.1.87.65:40985), 28eccbfd381040719fe51f2551bc4083 (127.1.87.66:35807)
I20260619 13:00:46.825503 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.823100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40985 (local address 127.1.87.67:39039)
0619 13:00:46.823387 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:46.823407 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:46.823542 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:46.823935 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:46.823940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:46.823963 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:46.824646 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:00:46.824650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:46.824673 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:00:46.824726 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:46.825084 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:46.825089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:46.825094 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:00:46.825222 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 13:00:46.825270 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 13:00:46.825539 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.823198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:39039 (local address 127.1.87.65:40985)
0619 13:00:46.823411 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.823418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.823494 (+ 76us) server_negotiation.cc:407] Connection header received
0619 13:00:46.823716 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.823721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.823783 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.823884 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.824872 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.824900 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.824907 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.824929 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.824966 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.824982 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.825099 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.825222 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.825266 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 13:00:46.826058 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.823152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:40291 (local address 127.1.87.66:35807)
0619 13:00:46.823379 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.823385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.824092 (+ 707us) server_negotiation.cc:407] Connection header received
0619 13:00:46.824260 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.824265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.824322 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.824420 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.825402 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.825410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.825414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.825431 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.825464 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.825478 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.825599 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.825762 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.825822 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:00:46.826573 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.823214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:35807 (local address 127.1.87.67:40291)
0619 13:00:46.823983 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:46.824004 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:46.824107 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:46.824510 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:46.824514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:46.824560 (+ 46us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:46.825204 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:00:46.825207 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:46.825222 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:00:46.825260 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:46.825584 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:46.825590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:46.825596 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:00:46.826252 (+ 656us) client_negotiation.cc:240] Negotiation successful
0619 13:00:46.826305 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":446,"threads_started":1}
I20260619 13:00:46.826524 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c659f9a6bd447889e39b5969dd52155" candidate_uuid: "b317e29a30a44aa7aa0288baf0f40117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" is_pre_election: true
I20260619 13:00:46.827018 6187 raft_consensus.cc:2454] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b317e29a30a44aa7aa0288baf0f40117 in term 0.
I20260619 13:00:46.827483 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c659f9a6bd447889e39b5969dd52155" candidate_uuid: "b317e29a30a44aa7aa0288baf0f40117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eccbfd381040719fe51f2551bc4083" is_pre_election: true
I20260619 13:00:46.827740 6384 leader_election.cc:302] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2404cfdf892b4c67b8ff8c59a0c796fd, b317e29a30a44aa7aa0288baf0f40117; no voters:
I20260619 13:00:46.827966 6318 raft_consensus.cc:2454] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b317e29a30a44aa7aa0288baf0f40117 in term 0.
I20260619 13:00:46.828109 6518 raft_consensus.cc:2781] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:00:46.828287 6518 raft_consensus.cc:482] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:00:46.828392 6518 raft_consensus.cc:3037] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:46.830493 6518 raft_consensus.cc:504] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } }
I20260619 13:00:46.831045 6518 leader_election.cc:288] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [CANDIDATE]: Term 1 election: Requested vote from peers 2404cfdf892b4c67b8ff8c59a0c796fd (127.1.87.65:40985), 28eccbfd381040719fe51f2551bc4083 (127.1.87.66:35807)
I20260619 13:00:46.831488 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c659f9a6bd447889e39b5969dd52155" candidate_uuid: "b317e29a30a44aa7aa0288baf0f40117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd"
I20260619 13:00:46.831501 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c659f9a6bd447889e39b5969dd52155" candidate_uuid: "b317e29a30a44aa7aa0288baf0f40117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eccbfd381040719fe51f2551bc4083"
I20260619 13:00:46.831656 6187 raft_consensus.cc:3037] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:46.831683 6318 raft_consensus.cc:3037] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:46.833696 6187 raft_consensus.cc:2454] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b317e29a30a44aa7aa0288baf0f40117 in term 1.
I20260619 13:00:46.833739 6318 raft_consensus.cc:2454] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b317e29a30a44aa7aa0288baf0f40117 in term 1.
I20260619 13:00:46.834077 6384 leader_election.cc:302] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2404cfdf892b4c67b8ff8c59a0c796fd, b317e29a30a44aa7aa0288baf0f40117; no voters:
I20260619 13:00:46.834342 6518 raft_consensus.cc:2781] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:46.834764 6518 raft_consensus.cc:686] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 1 LEADER]: Becoming Leader. State: Replica: b317e29a30a44aa7aa0288baf0f40117, State: Running, Role: LEADER
I20260619 13:00:46.835264 6518 consensus_queue.cc:237] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } }
I20260619 13:00:46.839709 6046 catalog_manager.cc:5495] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 reported cstate change: term changed from 0 to 1, leader changed from <none> to b317e29a30a44aa7aa0288baf0f40117 (127.1.87.67). New cstate: current_term: 1 leader_uuid: "b317e29a30a44aa7aa0288baf0f40117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b317e29a30a44aa7aa0288baf0f40117" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 } health_report { overall_health: UNKNOWN } } }
I20260619 13:00:46.874423 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.872418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.1.87.67:41865)
0619 13:00:46.872601 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.872607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.872691 (+ 84us) server_negotiation.cc:407] Connection header received
0619 13:00:46.872900 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.872905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.873022 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.873135 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.873948 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.873951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.873955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.873968 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.873997 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.874007 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.874105 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.874183 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.874223 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:00:46.882337 6318 raft_consensus.cc:1261] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [term 1 FOLLOWER]: Refusing update from remote peer b317e29a30a44aa7aa0288baf0f40117: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:46.882371 6187 raft_consensus.cc:1261] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [term 1 FOLLOWER]: Refusing update from remote peer b317e29a30a44aa7aa0288baf0f40117: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:00:46.883301 6518 consensus_queue.cc:1035] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eccbfd381040719fe51f2551bc4083" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:46.883630 6521 consensus_queue.cc:1035] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2404cfdf892b4c67b8ff8c59a0c796fd" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:46.909482 6046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:46.909751 6046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:00:46.913331 6046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42420bcaddbf458da39ad5bfa8ab309b: Sending DeleteTablet for 3 replicas of tablet 1c659f9a6bd447889e39b5969dd52155
I20260619 13:00:46.914417 6167 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c659f9a6bd447889e39b5969dd52155 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:46 UTC) from {username='slave'} at 127.0.0.1:50112
I20260619 13:00:46.914700 6298 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c659f9a6bd447889e39b5969dd52155 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:46 UTC) from {username='slave'} at 127.0.0.1:57768
I20260619 13:00:46.915118 6428 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c659f9a6bd447889e39b5969dd52155 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:00:46 UTC) from {username='slave'} at 127.0.0.1:60980
I20260619 13:00:46.916191 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 6105
I20260619 13:00:46.919361 6534 tablet_replica.cc:330] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: stopping tablet replica
I20260619 13:00:46.919932 6534 raft_consensus.cc:2229] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:46.920601 6534 raft_consensus.cc:2258] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:46.923158 6534 ts_tablet_manager.cc:1905] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:46.931680 6534 ts_tablet_manager.cc:1918] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:00:46.931874 6534 log.cc:1196] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/1c659f9a6bd447889e39b5969dd52155
I20260619 13:00:46.932425 6534 ts_tablet_manager.cc:1939] T 1c659f9a6bd447889e39b5969dd52155 P 28eccbfd381040719fe51f2551bc4083: Deleting consensus metadata
I20260619 13:00:46.934597 6032 catalog_manager.cc:4841] TS 28eccbfd381040719fe51f2551bc4083 (127.1.87.66:35807): tablet 1c659f9a6bd447889e39b5969dd52155 (table test-table [id=4dae901f1bfe438d80d3dcae60476ec7]) successfully deleted
I20260619 13:00:46.935710 6535 tablet_replica.cc:330] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: stopping tablet replica
I20260619 13:00:46.936407 6533 tablet_replica.cc:330] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd: stopping tablet replica
I20260619 13:00:46.936524 6535 raft_consensus.cc:2229] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:00:46.936792 6533 raft_consensus.cc:2229] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:46.937153 6535 raft_consensus.cc:2258] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:46.937318 6533 raft_consensus.cc:2258] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:46.938581 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:46.917530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.1.87.65:40985)
0619 13:00:46.917710 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:46.917717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:46.934908 (+ 17191us) server_negotiation.cc:407] Connection header received
0619 13:00:46.935111 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:46.935117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:46.935239 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:46.935374 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:46.937767 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:46.937773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:46.937778 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:46.937803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:46.937839 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:46.937854 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:46.937988 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:46.938218 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 13:00:46.938280 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:00:46.939599 6535 ts_tablet_manager.cc:1905] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:46.939694 6533 ts_tablet_manager.cc:1905] T 1c659f9a6bd447889e39b5969dd52155 P 2404cfdf892b4c67b8ff8c59a0c796fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:46.939844 6227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:46.949508 6535 ts_tablet_manager.cc:1918] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:00:46.949704 6535 log.cc:1196] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/1c659f9a6bd447889e39b5969dd52155
I20260619 13:00:46.950242 6535 ts_tablet_manager.cc:1939] T 1c659f9a6bd447889e39b5969dd52155 P b317e29a30a44aa7aa0288baf0f40117: Deleting consensus metadata
I20260619 13:00:46.952159 6033 catalog_manager.cc:4841] TS b317e29a30a44aa7aa0288baf0f40117 (127.1.87.67:41865): tablet 1c659f9a6bd447889e39b5969dd52155 (table test-table [id=4dae901f1bfe438d80d3dcae60476ec7]) successfully deleted
I20260619 13:00:47.036350 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 6105
I20260619 13:00:47.058832 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 6236
W20260619 13:00:47.058765 6033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.87.65:40985 (error 108)
I20260619 13:00:47.060684 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.060187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40985 (local address 127.0.0.1:50144)
0619 13:00:47.060381 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:47.060517 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 13:00:47.060963 6033 catalog_manager.cc:4584] TS 2404cfdf892b4c67b8ff8c59a0c796fd (127.1.87.65:40985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c659f9a6bd447889e39b5969dd52155: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40985: connect: Connection refused (error 111)
I20260619 13:00:47.062289 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.060217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.1.87.66:35807)
0619 13:00:47.060413 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:47.060419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:47.060437 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:00:47.060498 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:47.060503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:47.060601 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:47.060710 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:47.061676 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:47.061680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:47.061683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:47.061696 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:47.061731 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:47.061743 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:47.061837 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:47.061954 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 13:00:47.061997 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:00:47.062844 6358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:47.114614 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.114077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40985 (local address 127.0.0.1:50156)
0619 13:00:47.114301 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:47.114410 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 13:00:47.115777 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.115382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40985 (local address 127.0.0.1:50158)
0619 13:00:47.115534 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:47.115626 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 13:00:47.150058 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 6236
I20260619 13:00:47.166409 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 6366
I20260619 13:00:47.169615 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.167200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.1.87.67:41865)
0619 13:00:47.167452 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:47.167457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:47.167571 (+ 114us) server_negotiation.cc:407] Connection header received
0619 13:00:47.167785 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:47.167789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:47.167838 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:47.167917 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:47.168911 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:47.168915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:47.168918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:47.168933 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:47.168966 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:47.168977 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:47.169085 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:47.169216 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 13:00:47.169271 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:00:47.170257 6488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:47.176415 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.176000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40985 (local address 127.0.0.1:50170)
0619 13:00:47.176155 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:47.176273 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 13:00:47.177436 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.177140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40985 (local address 127.0.0.1:50176)
0619 13:00:47.177273 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:47.177335 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 13:00:47.258062 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.257573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40985 (local address 127.0.0.1:50184)
0619 13:00:47.257785 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:47.257896 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 13:00:47.259120 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.258744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:40985 (local address 127.0.0.1:50190)
0619 13:00:47.258947 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:47.259009 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 13:00:47.287211 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 6366
I20260619 13:00:47.305961 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 6015
I20260619 13:00:47.307049 6076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:47.409340 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 6015
2026-06-19T13:00:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2597 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 13:00:47.450451 1373 mini_oidc.cc:152] Starting JWKS server
I20260619 13:00:47.453114 1373 webserver.cc:460] Webserver started at https://127.0.0.1:46297/ using document root <none> and password file <none>
I20260619 13:00:47.453447 1373 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 13:00:47.454083 1373 webserver.cc:460] Webserver started at http://127.0.0.1:43027/ using document root <none> and password file <none>
2026-06-19T13:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:47Z Disabled control of system clock
I20260619 13:00:47.480159 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:34365
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:34365
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46297/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/testchainca.cert with env {}
W20260619 13:00:47.798410 6547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:47.798727 6547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:47.798816 6547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:47.806574 6547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:47.806685 6547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:47.806725 6547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:47.806774 6547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:47.806818 6547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:00:47.806879 6547 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:00:47.806918 6547 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46297/jwks.json
I20260619 13:00:47.817943 6547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:34365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:34365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46297/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:47.819217 6547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:47.820526 6547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:47.832635 6553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:47.832670 6555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:47.832698 6552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:47.833662 6547 server_base.cc:1034] running on GCE node
I20260619 13:00:47.834421 6547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:47.835898 6547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:47.837091 6547 hybrid_clock.cc:648] HybridClock initialized: now 1781874047837053 us; error 30 us; skew 500 ppm
I20260619 13:00:47.837450 6547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:00:47.839897 6547 webserver.cc:460] Webserver started at http://127.1.87.126:41067/ using document root <none> and password file <none>
I20260619 13:00:47.840598 6547 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:47.840693 6547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:47.841020 6547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:47.843962 6547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "f12671de1a5949089d51c80d813cc336"
format_stamp: "Formatted at 2026-06-19 13:00:47 on dist-test-slave-tpsm"
server_key: "d8c94a7700dfad18c57339c42a6ddad3"
server_key_iv: "b7ecdb0eda2282253087d102535cde6e"
server_key_version: "encryptionkey@0"
I20260619 13:00:47.844790 6547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "f12671de1a5949089d51c80d813cc336"
format_stamp: "Formatted at 2026-06-19 13:00:47 on dist-test-slave-tpsm"
server_key: "d8c94a7700dfad18c57339c42a6ddad3"
server_key_iv: "b7ecdb0eda2282253087d102535cde6e"
server_key_version: "encryptionkey@0"
I20260619 13:00:47.849949 6547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 13:00:47.853502 6561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:47.854887 6547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:47.855088 6547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "f12671de1a5949089d51c80d813cc336"
format_stamp: "Formatted at 2026-06-19 13:00:47 on dist-test-slave-tpsm"
server_key: "d8c94a7700dfad18c57339c42a6ddad3"
server_key_iv: "b7ecdb0eda2282253087d102535cde6e"
server_key_version: "encryptionkey@0"
I20260619 13:00:47.855325 6547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:47.872510 6547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:47.873389 6547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:47.873684 6547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:47.892414 6547 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:34365
I20260619 13:00:47.892437 6612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:34365 every 8 connection(s)
I20260619 13:00:47.893761 6547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:47.897400 6613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:47.898408 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 6547
I20260619 13:00:47.898636 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:47.899466 1373 external_mini_cluster.cc:1428] Setting key f2e3605d2af58732ef5913ee0047f0f9
I20260619 13:00:47.910598 6613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336: Bootstrap starting.
I20260619 13:00:47.912572 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:47.901380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.1.87.126:34365)
0619 13:00:47.902365 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:47.902376 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:47.902418 (+ 42us) server_negotiation.cc:407] Connection header received
0619 13:00:47.904069 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:47.904090 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:47.904548 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:47.904939 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:47.905487 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:47.906682 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:47.908054 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:47.908727 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:47.908933 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:47.910121 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:47.910128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:47.910141 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:47.910904 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:47.910980 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:47.911001 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:47.911139 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:47.911394 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 13:00:47.911680 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":361,"threads_started":1}
I20260619 13:00:47.914197 6613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:47.915349 6613 log.cc:824] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:47.916749 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:00:47.918216 6613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336: No bootstrap required, opened a new log
I20260619 13:00:47.926466 6613 raft_consensus.cc:348] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12671de1a5949089d51c80d813cc336" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34365 } }
I20260619 13:00:47.926875 6613 raft_consensus.cc:374] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:47.926988 6613 raft_consensus.cc:729] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12671de1a5949089d51c80d813cc336, State: Initialized, Role: FOLLOWER
I20260619 13:00:47.927619 6613 consensus_queue.cc:260] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12671de1a5949089d51c80d813cc336" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34365 } }
I20260619 13:00:47.927902 6613 raft_consensus.cc:388] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:47.928059 6613 raft_consensus.cc:482] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:47.928236 6613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:47.930922 6613 raft_consensus.cc:504] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12671de1a5949089d51c80d813cc336" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34365 } }
I20260619 13:00:47.931524 6613 leader_election.cc:302] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f12671de1a5949089d51c80d813cc336; no voters:
I20260619 13:00:47.932101 6613 leader_election.cc:288] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:47.932260 6618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:47.933025 6618 raft_consensus.cc:686] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [term 1 LEADER]: Becoming Leader. State: Replica: f12671de1a5949089d51c80d813cc336, State: Running, Role: LEADER
I20260619 13:00:47.933523 6618 consensus_queue.cc:237] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12671de1a5949089d51c80d813cc336" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34365 } }
I20260619 13:00:47.934237 6613 sys_catalog.cc:564] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:47.937028 6619 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f12671de1a5949089d51c80d813cc336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12671de1a5949089d51c80d813cc336" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34365 } } }
I20260619 13:00:47.937582 6619 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:47.938547 6620 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f12671de1a5949089d51c80d813cc336. Latest consensus state: current_term: 1 leader_uuid: "f12671de1a5949089d51c80d813cc336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12671de1a5949089d51c80d813cc336" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34365 } } }
I20260619 13:00:47.938947 6620 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:47.939106 6628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:47.944062 6628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:47.951719 6628 catalog_manager.cc:1347] Generated new cluster ID: dfcd9a881a0e4f08be081d27948f6d4b
I20260619 13:00:47.951838 6628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:47.973960 6628 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:47.975826 6628 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:47.995391 6628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f12671de1a5949089d51c80d813cc336: Generated new TSK 0
I20260619 13:00:47.996261 6628 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 13:00:48.033469 6643 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 13:00:48.033643 6643 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 13:00:48.034875 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:48.013148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.1.87.126:34365)
0619 13:00:48.013349 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:48.013356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:48.013972 (+ 616us) server_negotiation.cc:407] Connection header received
0619 13:00:48.014244 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:48.014250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:48.014343 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:48.014497 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:00:48.014972 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.016170 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.017562 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.018319 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.018451 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:48.018963 (+ 512us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:00:48.034170 (+ 15207us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 13:00:48.034568 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52192: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 13:00:48.035003 6616 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52192: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 13:00:48.036420 6642 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:00:48.012984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:34365 (local address 127.0.0.1:52192)
0619 13:00:48.013876 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:48.013892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:48.014029 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:48.014523 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:48.014526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:48.014534 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:00:48.014839 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:48.014847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.016298 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.016302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:48.017416 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:48.017426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.018489 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.018494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:48.018612 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:48.018679 (+ 67us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:00:48.034521 (+ 15842us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 13:00:48.034799 (+ 278us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.87.126:34365: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":573,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.87.126:34365: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 13:00:48.038383 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 6547
I20260619 13:00:48.039655 6608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:48.132965 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 6547
2026-06-19T13:00:48Z chronyd exiting
I20260619 13:00:48.183077 1373 test_util.cc:175] -----------------------------------------------
I20260619 13:00:48.183223 1373 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874004469744-1373-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (737 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 13:00:48.186933 1373 mini_oidc.cc:152] Starting JWKS server
I20260619 13:00:48.188854 1373 webserver.cc:460] Webserver started at https://127.0.0.1:44141/ using document root <none> and password file <none>
I20260619 13:00:48.189085 1373 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 13:00:48.189749 1373 webserver.cc:460] Webserver started at http://127.0.0.1:44387/ using document root <none> and password file <none>
2026-06-19T13:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:48Z Disabled control of system clock
I20260619 13:00:48.216279 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39587
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:39587
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44141/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/testchainca.cert with env {}
W20260619 13:00:48.532706 6650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:48.532975 6650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:48.533120 6650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:48.540237 6650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:48.540375 6650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:48.540446 6650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:48.540525 6650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:48.540591 6650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:00:48.540673 6650 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:00:48.540752 6650 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44141/jwks.json
I20260619 13:00:48.550701 6650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:39587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44141/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:48.552028 6650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:48.553229 6650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:48.564320 6658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:48.564355 6656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:48.564401 6655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:48.565111 6650 server_base.cc:1034] running on GCE node
I20260619 13:00:48.565840 6650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:48.567137 6650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:48.568359 6650 hybrid_clock.cc:648] HybridClock initialized: now 1781874048568316 us; error 33 us; skew 500 ppm
I20260619 13:00:48.568699 6650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:00:48.570801 6650 webserver.cc:460] Webserver started at http://127.1.87.126:37919/ using document root <none> and password file <none>
I20260619 13:00:48.571520 6650 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:48.571650 6650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:48.571964 6650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:48.574643 6650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "34810578a3e94bc9a9d449c623bee4a6"
format_stamp: "Formatted at 2026-06-19 13:00:48 on dist-test-slave-tpsm"
server_key: "3f475d2487f8a13a39666d3434d45064"
server_key_iv: "7a968c854d342cb6521359c1a48bd2b2"
server_key_version: "encryptionkey@0"
I20260619 13:00:48.575464 6650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "34810578a3e94bc9a9d449c623bee4a6"
format_stamp: "Formatted at 2026-06-19 13:00:48 on dist-test-slave-tpsm"
server_key: "3f475d2487f8a13a39666d3434d45064"
server_key_iv: "7a968c854d342cb6521359c1a48bd2b2"
server_key_version: "encryptionkey@0"
I20260619 13:00:48.580488 6650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:00:48.583731 6664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:48.585026 6650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:00:48.585211 6650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "34810578a3e94bc9a9d449c623bee4a6"
format_stamp: "Formatted at 2026-06-19 13:00:48 on dist-test-slave-tpsm"
server_key: "3f475d2487f8a13a39666d3434d45064"
server_key_iv: "7a968c854d342cb6521359c1a48bd2b2"
server_key_version: "encryptionkey@0"
I20260619 13:00:48.585433 6650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:48.611462 6650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:48.612257 6650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:48.612538 6650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:48.630020 6650 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:39587
I20260619 13:00:48.630030 6715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:39587 every 8 connection(s)
I20260619 13:00:48.631520 6650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:48.633950 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 6650
I20260619 13:00:48.634088 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:48.634696 1373 external_mini_cluster.cc:1428] Setting key 156d770eadd28b10134c471e1efe7a4e
I20260619 13:00:48.635694 6716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:48.649708 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:48.636747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.1.87.126:39587)
0619 13:00:48.637866 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:48.637880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:48.638005 (+ 125us) server_negotiation.cc:407] Connection header received
0619 13:00:48.640121 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:48.640146 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:48.640682 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:48.641099 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:48.641582 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.643547 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.644790 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.645785 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.645960 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:48.647015 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:48.647025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:48.647042 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:48.647771 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:48.647862 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:48.647886 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:48.648001 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:48.648226 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 13:00:48.648507 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":427,"threads_started":1}
I20260619 13:00:48.650117 6716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6: Bootstrap starting.
I20260619 13:00:48.653194 6716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:48.653385 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:00:48.654168 6716 log.cc:824] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:48.656711 6716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6: No bootstrap required, opened a new log
I20260619 13:00:48.663290 6716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34810578a3e94bc9a9d449c623bee4a6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39587 } }
I20260619 13:00:48.663687 6716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:48.663792 6716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34810578a3e94bc9a9d449c623bee4a6, State: Initialized, Role: FOLLOWER
I20260619 13:00:48.664402 6716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34810578a3e94bc9a9d449c623bee4a6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39587 } }
I20260619 13:00:48.664667 6716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:48.664772 6716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:48.664885 6716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:48.666781 6716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34810578a3e94bc9a9d449c623bee4a6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39587 } }
I20260619 13:00:48.667178 6716 leader_election.cc:302] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34810578a3e94bc9a9d449c623bee4a6; no voters:
I20260619 13:00:48.667780 6716 leader_election.cc:288] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:48.667919 6721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:48.668567 6721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [term 1 LEADER]: Becoming Leader. State: Replica: 34810578a3e94bc9a9d449c623bee4a6, State: Running, Role: LEADER
I20260619 13:00:48.669147 6721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34810578a3e94bc9a9d449c623bee4a6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39587 } }
I20260619 13:00:48.669900 6716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:48.671904 6723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34810578a3e94bc9a9d449c623bee4a6. Latest consensus state: current_term: 1 leader_uuid: "34810578a3e94bc9a9d449c623bee4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34810578a3e94bc9a9d449c623bee4a6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39587 } } }
I20260619 13:00:48.672339 6723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:48.672830 6722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34810578a3e94bc9a9d449c623bee4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34810578a3e94bc9a9d449c623bee4a6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39587 } } }
I20260619 13:00:48.673290 6722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:48.677153 6730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:48.681583 6730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:48.689906 6730 catalog_manager.cc:1347] Generated new cluster ID: b22d72b9afef4fd9bedfda8c7b428b56
I20260619 13:00:48.690022 6730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:48.707825 6730 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:48.709584 6730 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:48.720896 6730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34810578a3e94bc9a9d449c623bee4a6: Generated new TSK 0
I20260619 13:00:48.721971 6730 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 13:00:48.763998 6746 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 13:00:48.764148 6746 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260619 13:00:48.765429 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:00:48.743198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39587 (local address 127.0.0.1:36758)
0619 13:00:48.743850 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:48.743870 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:48.744013 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:48.744438 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:48.744442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:48.744454 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:00:48.744755 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:48.744767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.745783 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.745789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:48.747496 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:48.747510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.748265 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.748275 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:48.748499 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:48.748550 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:00:48.764946 (+ 16396us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 13:00:48.765142 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.87.126:39587: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":362,"threads_started":1}
I20260619 13:00:48.765471 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:48.743255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.1.87.126:39587)
0619 13:00:48.743499 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:48.743508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:48.743975 (+ 467us) server_negotiation.cc:407] Connection header received
0619 13:00:48.744168 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:48.744172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:48.744263 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:48.744399 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:00:48.744919 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.745650 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.747633 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:48.748084 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:48.748184 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:48.748791 (+ 607us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:00:48.764710 (+ 15919us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 13:00:48.765124 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 13:00:48.765591 6719 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 13:00:48.766140 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 6650
I20260619 13:00:48.767406 6711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:48.857831 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 6650
2026-06-19T13:00:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (725 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 13:00:48.912694 1373 mini_oidc.cc:152] Starting JWKS server
I20260619 13:00:48.914578 1373 webserver.cc:460] Webserver started at https://127.0.0.1:42205/ using document root <none> and password file <none>
I20260619 13:00:48.914822 1373 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 13:00:48.915491 1373 webserver.cc:460] Webserver started at http://127.0.0.1:33273/ using document root <none> and password file <none>
2026-06-19T13:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:48Z Disabled control of system clock
I20260619 13:00:48.940632 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:35403
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:35403
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42205/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 13:00:49.250697 6753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:49.251039 6753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:49.251216 6753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:49.258553 6753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:49.258699 6753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:49.258795 6753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:49.258908 6753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:49.258992 6753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:00:49.259081 6753 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:00:49.259160 6753 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42205/jwks.json
I20260619 13:00:49.268527 6753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:35403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:35403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42205/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:49.269873 6753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:49.271608 6753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:49.283165 6761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:49.283288 6758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:49.283257 6759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:49.283974 6753 server_base.cc:1034] running on GCE node
I20260619 13:00:49.284615 6753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:49.285861 6753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:49.287086 6753 hybrid_clock.cc:648] HybridClock initialized: now 1781874049287048 us; error 60 us; skew 500 ppm
I20260619 13:00:49.287483 6753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:00:49.289608 6753 webserver.cc:460] Webserver started at http://127.1.87.126:35527/ using document root <none> and password file <none>
I20260619 13:00:49.290275 6753 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:49.290403 6753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:49.290712 6753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:49.293519 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "211eeade5d4044fa88d4200daba5f887"
format_stamp: "Formatted at 2026-06-19 13:00:49 on dist-test-slave-tpsm"
server_key: "3c5abd19fcfc9e0335bb26293eb66081"
server_key_iv: "4c39881e54d721227e8e5ed2c4c8f3ab"
server_key_version: "encryptionkey@0"
I20260619 13:00:49.294298 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "211eeade5d4044fa88d4200daba5f887"
format_stamp: "Formatted at 2026-06-19 13:00:49 on dist-test-slave-tpsm"
server_key: "3c5abd19fcfc9e0335bb26293eb66081"
server_key_iv: "4c39881e54d721227e8e5ed2c4c8f3ab"
server_key_version: "encryptionkey@0"
I20260619 13:00:49.299126 6753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 13:00:49.302361 6767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:49.303705 6753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:49.303884 6753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "211eeade5d4044fa88d4200daba5f887"
format_stamp: "Formatted at 2026-06-19 13:00:49 on dist-test-slave-tpsm"
server_key: "3c5abd19fcfc9e0335bb26293eb66081"
server_key_iv: "4c39881e54d721227e8e5ed2c4c8f3ab"
server_key_version: "encryptionkey@0"
I20260619 13:00:49.304103 6753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:49.329298 6753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:49.330088 6753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:49.330372 6753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:49.348196 6753 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:35403
I20260619 13:00:49.348219 6818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:35403 every 8 connection(s)
I20260619 13:00:49.349488 6753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:49.353081 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:49.357700 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 6753
I20260619 13:00:49.357920 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:49.358527 1373 external_mini_cluster.cc:1428] Setting key 16709733d6d6b4291f910c03149c4aab
I20260619 13:00:49.364512 6819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887: Bootstrap starting.
I20260619 13:00:49.368223 6819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:49.369405 6819 log.cc:824] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:49.372691 6819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887: No bootstrap required, opened a new log
I20260619 13:00:49.373756 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:49.360463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.1.87.126:35403)
0619 13:00:49.361555 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:49.361569 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:49.361616 (+ 47us) server_negotiation.cc:407] Connection header received
0619 13:00:49.363900 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:49.363926 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:49.364470 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:49.364889 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:49.365374 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:49.367420 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:49.368752 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:49.369784 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:49.369997 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:49.371193 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:49.371205 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:49.371222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:49.371907 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:49.371995 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:49.372019 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:49.372160 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:49.372280 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 13:00:49.372578 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":358,"threads_started":1}
I20260619 13:00:49.377363 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:00:49.380954 6819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211eeade5d4044fa88d4200daba5f887" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35403 } }
I20260619 13:00:49.381278 6819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:49.381388 6819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211eeade5d4044fa88d4200daba5f887, State: Initialized, Role: FOLLOWER
I20260619 13:00:49.381901 6819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211eeade5d4044fa88d4200daba5f887" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35403 } }
I20260619 13:00:49.382123 6819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:49.382263 6819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:49.382428 6819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:49.384807 6819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211eeade5d4044fa88d4200daba5f887" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35403 } }
I20260619 13:00:49.385310 6819 leader_election.cc:302] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211eeade5d4044fa88d4200daba5f887; no voters:
I20260619 13:00:49.385843 6819 leader_election.cc:288] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:49.385993 6824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:49.386673 6824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [term 1 LEADER]: Becoming Leader. State: Replica: 211eeade5d4044fa88d4200daba5f887, State: Running, Role: LEADER
I20260619 13:00:49.387279 6824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211eeade5d4044fa88d4200daba5f887" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35403 } }
I20260619 13:00:49.387845 6819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:49.390333 6825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211eeade5d4044fa88d4200daba5f887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211eeade5d4044fa88d4200daba5f887" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35403 } } }
I20260619 13:00:49.390368 6826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211eeade5d4044fa88d4200daba5f887. Latest consensus state: current_term: 1 leader_uuid: "211eeade5d4044fa88d4200daba5f887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211eeade5d4044fa88d4200daba5f887" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35403 } } }
I20260619 13:00:49.390805 6825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:49.390851 6826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:49.391736 6833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:49.398243 6833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:49.407410 6833 catalog_manager.cc:1347] Generated new cluster ID: 7aace213bc784c59a9a04cae80bd0488
I20260619 13:00:49.407563 6833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:49.426465 6833 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:49.428290 6833 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:49.435997 6833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 211eeade5d4044fa88d4200daba5f887: Generated new TSK 0
I20260619 13:00:49.436920 6833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:00:49.465442 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:49.456526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.1.87.126:35403)
0619 13:00:49.456764 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:49.456770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:49.457130 (+ 360us) server_negotiation.cc:407] Connection header received
0619 13:00:49.457433 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:49.457438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:49.457514 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:49.457672 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:00:49.458214 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:49.458981 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:49.461028 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:49.461764 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:49.461967 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:49.462641 (+ 674us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:00:49.464727 (+ 2086us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 13:00:49.465092 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42714: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":103}
W20260619 13:00:49.465595 6848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:00:49.456324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:35403 (local address 127.0.0.1:42714)
0619 13:00:49.457010 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:49.457031 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:49.457199 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:49.457708 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:49.457714 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:49.457726 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:00:49.458029 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:49.458045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:49.459129 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:49.459135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:49.460854 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:49.460870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:49.462042 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:49.462049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:49.462266 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:49.462314 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:00:49.464938 (+ 2624us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 13:00:49.465193 (+ 255us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.87.126:35403: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":340,"threads_started":1}
W20260619 13:00:49.465942 6822 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42714: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260619 13:00:49.467767 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 6753
I20260619 13:00:49.469002 6814 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 13:00:49.478518 6849 webserver.cc:533] Webserver: sslize failed:
E20260619 13:00:49.478658 6849 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 13:00:49.562565 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 6753
2026-06-19T13:00:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (704 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T13:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:49Z Disabled control of system clock
I20260619 13:00:49.646770 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44169
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:44169
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/testchainca.cert with env {}
W20260619 13:00:49.964986 6854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:49.965291 6854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:49.965468 6854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:49.972720 6854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:49.972822 6854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:49.972929 6854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:49.973024 6854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:49.973109 6854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:00:49.973209 6854 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:00:49.973264 6854 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 13:00:49.983300 6854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:44169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:49.984681 6854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:49.985937 6854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:49.997813 6859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:49.997898 6862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:49.997786 6860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:49.998728 6854 server_base.cc:1034] running on GCE node
I20260619 13:00:49.999574 6854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:50.001031 6854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:50.002262 6854 hybrid_clock.cc:648] HybridClock initialized: now 1781874050002227 us; error 65 us; skew 500 ppm
I20260619 13:00:50.002615 6854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:00:50.005111 6854 webserver.cc:460] Webserver started at http://127.1.87.126:41797/ using document root <none> and password file <none>
I20260619 13:00:50.005815 6854 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:50.005950 6854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:50.006271 6854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:50.009100 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de"
format_stamp: "Formatted at 2026-06-19 13:00:50 on dist-test-slave-tpsm"
server_key: "82bd35b4460b784bd597cd6606fbc602"
server_key_iv: "8449dfe5a2efad2623f0d4b4a3b39a18"
server_key_version: "encryptionkey@0"
I20260619 13:00:50.009902 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de"
format_stamp: "Formatted at 2026-06-19 13:00:50 on dist-test-slave-tpsm"
server_key: "82bd35b4460b784bd597cd6606fbc602"
server_key_iv: "8449dfe5a2efad2623f0d4b4a3b39a18"
server_key_version: "encryptionkey@0"
I20260619 13:00:50.014786 6854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:00:50.018142 6868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:50.019541 6854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:00:50.019706 6854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de"
format_stamp: "Formatted at 2026-06-19 13:00:50 on dist-test-slave-tpsm"
server_key: "82bd35b4460b784bd597cd6606fbc602"
server_key_iv: "8449dfe5a2efad2623f0d4b4a3b39a18"
server_key_version: "encryptionkey@0"
I20260619 13:00:50.019896 6854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:50.033337 6854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:50.034089 6854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:50.034368 6854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:50.052786 6854 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:44169
I20260619 13:00:50.052810 6919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:44169 every 8 connection(s)
I20260619 13:00:50.054140 6854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:50.056126 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 6854
I20260619 13:00:50.056316 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:50.057099 1373 external_mini_cluster.cc:1428] Setting key a8971f9e6c215261ffbde74c2cd1ec28
I20260619 13:00:50.058563 6920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:50.070076 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:50.059013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.1.87.126:44169)
0619 13:00:50.059946 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:50.059956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:50.059996 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:00:50.061463 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:50.061481 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:50.061984 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:50.062357 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:50.062965 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:50.064453 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:50.065685 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:50.066341 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:50.066540 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:50.067817 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:50.067825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:50.067838 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:00:50.068396 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:50.068471 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:00:50.068490 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:50.068624 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:50.068840 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 13:00:50.069132 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":352,"threads_started":1}
I20260619 13:00:50.071810 6920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de: Bootstrap starting.
I20260619 13:00:50.073968 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:00:50.075467 6920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:50.076553 6920 log.cc:824] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:50.079185 6920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de: No bootstrap required, opened a new log
I20260619 13:00:50.086324 6920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44169 } }
I20260619 13:00:50.086687 6920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:50.086805 6920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9de3b3123a4bea9b2cf5d1934bf9de, State: Initialized, Role: FOLLOWER
I20260619 13:00:50.087430 6920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44169 } }
I20260619 13:00:50.087634 6920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:50.087808 6920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:50.087966 6920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:50.090075 6920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44169 } }
I20260619 13:00:50.090587 6920 leader_election.cc:302] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a9de3b3123a4bea9b2cf5d1934bf9de; no voters:
I20260619 13:00:50.091111 6920 leader_election.cc:288] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:50.091292 6925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:50.092103 6925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [term 1 LEADER]: Becoming Leader. State: Replica: 9a9de3b3123a4bea9b2cf5d1934bf9de, State: Running, Role: LEADER
I20260619 13:00:50.092687 6925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44169 } }
I20260619 13:00:50.093241 6920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:50.095193 6927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9de3b3123a4bea9b2cf5d1934bf9de. Latest consensus state: current_term: 1 leader_uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44169 } } }
I20260619 13:00:50.095439 6926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9de3b3123a4bea9b2cf5d1934bf9de" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44169 } } }
I20260619 13:00:50.095675 6927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:50.095717 6926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:50.097651 6930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:50.102792 6930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:50.111470 6930 catalog_manager.cc:1347] Generated new cluster ID: 54057832def1427c891bb15da2ad42ed
I20260619 13:00:50.111653 6930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:50.146526 6930 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:50.148463 6930 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:50.155967 6930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a9de3b3123a4bea9b2cf5d1934bf9de: Generated new TSK 0
I20260619 13:00:50.156942 6930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:00:50.225831 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:50.175549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.1.87.126:44169)
0619 13:00:50.175793 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:50.175802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:50.176207 (+ 405us) server_negotiation.cc:407] Connection header received
0619 13:00:50.176434 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:50.176438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:50.176506 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:50.176629 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:00:50.177178 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:50.177879 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:50.179871 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:50.180340 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:50.180439 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:50.180973 (+ 534us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:00:50.225125 (+ 44152us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 13:00:50.225522 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50130: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 13:00:50.225966 6923 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50130: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 13:00:50.226121 6949 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:00:50.175494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44169 (local address 127.0.0.1:50130)
0619 13:00:50.176089 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:50.176108 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:50.176255 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:50.176671 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:50.176683 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:50.176696 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:00:50.177025 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:50.177036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:50.178017 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:50.178023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:50.179731 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:50.179745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:50.180510 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:50.180516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:50.180700 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:50.180741 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:00:50.225451 (+ 44710us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 13:00:50.225704 (+ 253us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.87.126:44169: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":328,"threads_started":1}
I20260619 13:00:50.227787 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 6854
I20260619 13:00:50.229023 6915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:50.325512 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 6854
2026-06-19T13:00:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (742 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (411 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (404 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:51 dist-test-slave-tpsm krb5kdc[6964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:51 dist-test-slave-tpsm krb5kdc[6964](info): set up 2 sockets
Jun 19 13:00:51 dist-test-slave-tpsm krb5kdc[6964](info): commencing operation
krb5kdc: starting...
W20260619 13:00:52.220079 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:52 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:00:52.450289 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:43789
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:43789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:52.781818 6977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:52.782232 6977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:52.782619 6977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:52.791625 6977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:52.791762 6977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:52.791872 6977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:52.792007 6977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:52.792078 6977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:52.803249 6977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:43789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:43789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:52.804656 6977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:52.805935 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:52.817829 6982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:52.817768 6983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:52.817801 6985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:52.818908 6977 server_base.cc:1034] running on GCE node
I20260619 13:00:52.819821 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:52.821221 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:52.822432 6977 hybrid_clock.cc:648] HybridClock initialized: now 1781874052822383 us; error 39 us; skew 500 ppm
I20260619 13:00:52.822799 6977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:52 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874052, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:52.826036 6977 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:00:52.827528 6977 webserver.cc:460] Webserver started at http://127.1.87.126:42741/ using document root <none> and password file <none>
I20260619 13:00:52.828191 6977 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:52.828318 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:52.828639 6977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:52.831624 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "2a8873e4772146b0b219294fae65eff0"
format_stamp: "Formatted at 2026-06-19 13:00:52 on dist-test-slave-tpsm"
server_key: "673d431227d878d567cf2b421b68bcc7"
server_key_iv: "aa6b16824b8f203636ac967d5549f9a4"
server_key_version: "encryptionkey@0"
I20260619 13:00:52.832372 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "2a8873e4772146b0b219294fae65eff0"
format_stamp: "Formatted at 2026-06-19 13:00:52 on dist-test-slave-tpsm"
server_key: "673d431227d878d567cf2b421b68bcc7"
server_key_iv: "aa6b16824b8f203636ac967d5549f9a4"
server_key_version: "encryptionkey@0"
I20260619 13:00:52.837365 6977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:00:52.840683 6992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:52.841962 6977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:00:52.842168 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "2a8873e4772146b0b219294fae65eff0"
format_stamp: "Formatted at 2026-06-19 13:00:52 on dist-test-slave-tpsm"
server_key: "673d431227d878d567cf2b421b68bcc7"
server_key_iv: "aa6b16824b8f203636ac967d5549f9a4"
server_key_version: "encryptionkey@0"
I20260619 13:00:52.842371 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:52.855181 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:52.859387 6977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:52.859663 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:52.878115 6977 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:43789
I20260619 13:00:52.878204 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:43789 every 8 connection(s)
I20260619 13:00:52.879498 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:52.880872 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 6977
I20260619 13:00:52.881069 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:52.881857 1373 external_mini_cluster.cc:1428] Setting key 4d1769380df252ff4de50168314296ed
I20260619 13:00:52.884369 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:00:52 dist-test-slave-tpsm krb5kdc[6964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:52.897939 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0: Bootstrap starting.
I20260619 13:00:52.901147 7044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:52.902117 7044 log.cc:824] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:52.904902 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0: No bootstrap required, opened a new log
I20260619 13:00:52.904985 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:52.884268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.1.87.126:43789)
0619 13:00:52.885068 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:52.885079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:52.885118 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:00:52.886745 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:52.886764 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:52.887109 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:52.887517 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:52.888891 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:52.890394 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:52.892226 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:52.892873 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:52.893002 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:52.897630 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:52.897638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:52.897652 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:52.897697 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:52.900911 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:52.901714 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:52.901721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:52.901726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:52.901894 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:52.902316 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:52.902321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:52.902324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:52.902851 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:52.903215 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:52.903619 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 13:00:52.904033 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":358,"threads_started":1}
I20260619 13:00:52.912961 7044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8873e4772146b0b219294fae65eff0" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43789 } }
I20260619 13:00:52.913364 7044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:52.913471 7044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8873e4772146b0b219294fae65eff0, State: Initialized, Role: FOLLOWER
I20260619 13:00:52.914031 7044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8873e4772146b0b219294fae65eff0" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43789 } }
I20260619 13:00:52.914270 7044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:52.914410 7044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:52.914594 7044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:52.917037 7044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8873e4772146b0b219294fae65eff0" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43789 } }
I20260619 13:00:52.917532 7044 leader_election.cc:302] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a8873e4772146b0b219294fae65eff0; no voters:
I20260619 13:00:52.918129 7044 leader_election.cc:288] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:52.918314 7049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:52.918984 7049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a8873e4772146b0b219294fae65eff0, State: Running, Role: LEADER
I20260619 13:00:52.919543 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8873e4772146b0b219294fae65eff0" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43789 } }
I20260619 13:00:52.920215 7044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:52.922224 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a8873e4772146b0b219294fae65eff0. Latest consensus state: current_term: 1 leader_uuid: "2a8873e4772146b0b219294fae65eff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8873e4772146b0b219294fae65eff0" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43789 } } }
I20260619 13:00:52.922569 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:52.923377 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a8873e4772146b0b219294fae65eff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8873e4772146b0b219294fae65eff0" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43789 } } }
I20260619 13:00:52.923707 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:52.928092 7055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:52.933238 7055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:52.941147 7055 catalog_manager.cc:1347] Generated new cluster ID: 876597d4ca5b440c8c74d21a4dc500a8
I20260619 13:00:52.941272 7055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:52.967236 7055 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:52.968837 7055 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:52.978973 7055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a8873e4772146b0b219294fae65eff0: Generated new TSK 0
I20260619 13:00:52.979899 7055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:00:53.038475 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43789
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:53.356200 7070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:53.356532 7070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:53.356628 7070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:53.364058 7070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:53.364190 7070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:53.364302 7070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:53.374264 7070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:53.375561 7070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:53.376950 7070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:53.390462 7076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:53.390583 7075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:53.390553 7078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:53.391568 7070 server_base.cc:1034] running on GCE node
I20260619 13:00:53.392187 7070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:53.393114 7070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:53.394356 7070 hybrid_clock.cc:648] HybridClock initialized: now 1781874053394276 us; error 57 us; skew 500 ppm
I20260619 13:00:53.394748 7070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:53 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874053, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:53.399000 7070 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:00:53.400863 7070 webserver.cc:460] Webserver started at http://127.1.87.65:41351/ using document root <none> and password file <none>
I20260619 13:00:53.401602 7070 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:53.401753 7070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:53.402065 7070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:53.405058 7070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "8c627852b9d64201972661c529e68759"
format_stamp: "Formatted at 2026-06-19 13:00:53 on dist-test-slave-tpsm"
server_key: "81a46187bc6db52187f26baa403b8947"
server_key_iv: "140fc6417bd312e2a460e41f3d7d4e70"
server_key_version: "encryptionkey@0"
I20260619 13:00:53.405853 7070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "8c627852b9d64201972661c529e68759"
format_stamp: "Formatted at 2026-06-19 13:00:53 on dist-test-slave-tpsm"
server_key: "81a46187bc6db52187f26baa403b8947"
server_key_iv: "140fc6417bd312e2a460e41f3d7d4e70"
server_key_version: "encryptionkey@0"
I20260619 13:00:53.411577 7070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:00:53.415086 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:53.416535 7070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:53.416723 7070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "8c627852b9d64201972661c529e68759"
format_stamp: "Formatted at 2026-06-19 13:00:53 on dist-test-slave-tpsm"
server_key: "81a46187bc6db52187f26baa403b8947"
server_key_iv: "140fc6417bd312e2a460e41f3d7d4e70"
server_key_version: "encryptionkey@0"
I20260619 13:00:53.416937 7070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:53.439596 7070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:53.443707 7070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:53.444002 7070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:53.444960 7070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:53.446373 7070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:53.446475 7070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:53.446619 7070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:53.446688 7070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:53.479207 7070 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:34773
I20260619 13:00:53.479226 7197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:34773 every 8 connection(s)
I20260619 13:00:53.480484 7070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:53.488984 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 7070
I20260619 13:00:53.489197 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:53.490010 1373 external_mini_cluster.cc:1428] Setting key ab8e4bad96479f0badd841806a11a36d
Jun 19 13:00:53 dist-test-slave-tpsm krb5kdc[6964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874053, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:53.500656 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:53.483026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:48209 (local address 127.1.87.126:43789)
0619 13:00:53.483560 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:53.483567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:53.484538 (+ 971us) server_negotiation.cc:407] Connection header received
0619 13:00:53.486236 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:53.486240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:53.486317 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:53.486428 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:53.487945 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:53.488692 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:53.490934 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:53.491492 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:53.491632 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:53.495075 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:53.495080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:53.495083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:53.495121 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:53.497670 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:53.498380 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:53.498386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:53.498390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:53.498504 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:53.498930 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:53.498935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:53.498939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:53.499197 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:53.499378 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:53.500056 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 13:00:53.500287 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":418,"thread_start_us":214,"threads_started":1}
I20260619 13:00:53.501622 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:53.483694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:43789 (local address 127.1.87.65:48209)
0619 13:00:53.484385 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:53.484427 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:53.485988 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:53.486644 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:53.486661 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:53.487159 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:53.487785 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:53.487805 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:53.488854 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:53.488861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:53.490798 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:53.490810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:53.491741 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:53.491750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:53.491987 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:53.492680 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:53.492703 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:53.494842 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:53.497817 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:53.497827 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:53.497845 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:53.498258 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:53.498624 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:53.498628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:53.498632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:53.498810 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:53.499348 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:53.499357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:53.499699 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 13:00:53.499966 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 13:00:53.500502 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":275,"threads_started":1}
I20260619 13:00:53.503237 7198 heartbeater.cc:347] Connected to a master server at 127.1.87.126:43789
I20260619 13:00:53.503571 7198 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:53.504441 7198 heartbeater.cc:511] Master 127.1.87.126:43789 requested a full tablet report, sending...
I20260619 13:00:53.506968 7009 ts_manager.cc:194] Registered new tserver with Master: 8c627852b9d64201972661c529e68759 (127.1.87.65:34773)
I20260619 13:00:53.509167 7009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:48209
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:00:53.536770 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43789
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:53.854641 7204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:53.854955 7204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:53.855085 7204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:53.862540 7204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:53.862684 7204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:53.862857 7204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:53.872939 7204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:53.874239 7204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:53.875532 7204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:53.894932 7210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:53.894932 7209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:53.895226 7212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:53.897600 7204 server_base.cc:1034] running on GCE node
I20260619 13:00:53.898249 7204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:53.899291 7204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:53.900615 7204 hybrid_clock.cc:648] HybridClock initialized: now 1781874053900564 us; error 39 us; skew 500 ppm
I20260619 13:00:53.900961 7204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:53 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874053, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:53.904158 7204 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:00:53.905673 7204 webserver.cc:460] Webserver started at http://127.1.87.66:45125/ using document root <none> and password file <none>
I20260619 13:00:53.906365 7204 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:53.906507 7204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:53.906813 7204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:53.909796 7204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "e8bb64e5c70f4156ae503a277577fa13"
format_stamp: "Formatted at 2026-06-19 13:00:53 on dist-test-slave-tpsm"
server_key: "f08c98ba04bf3a413b7f820868310ddd"
server_key_iv: "d408f8cde018a11cc6c46fe5dc479afd"
server_key_version: "encryptionkey@0"
I20260619 13:00:53.910557 7204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "e8bb64e5c70f4156ae503a277577fa13"
format_stamp: "Formatted at 2026-06-19 13:00:53 on dist-test-slave-tpsm"
server_key: "f08c98ba04bf3a413b7f820868310ddd"
server_key_iv: "d408f8cde018a11cc6c46fe5dc479afd"
server_key_version: "encryptionkey@0"
I20260619 13:00:53.916003 7204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 13:00:53.919435 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:53.920817 7204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:00:53.921016 7204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "e8bb64e5c70f4156ae503a277577fa13"
format_stamp: "Formatted at 2026-06-19 13:00:53 on dist-test-slave-tpsm"
server_key: "f08c98ba04bf3a413b7f820868310ddd"
server_key_iv: "d408f8cde018a11cc6c46fe5dc479afd"
server_key_version: "encryptionkey@0"
I20260619 13:00:53.921234 7204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:53.948879 7204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:53.953261 7204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:53.953562 7204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:53.954547 7204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:53.955888 7204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:53.955961 7204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:53.956091 7204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:53.956141 7204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:53.988059 7204 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:38379
I20260619 13:00:53.988114 7331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:38379 every 8 connection(s)
I20260619 13:00:53.989269 7204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:53.997931 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 7204
I20260619 13:00:53.998114 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:53.998580 1373 external_mini_cluster.cc:1428] Setting key daa6b2902e95106b1155a822421b27f7
Jun 19 13:00:53 dist-test-slave-tpsm krb5kdc[6964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874053, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:54.010296 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:53.991751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:49283 (local address 127.1.87.126:43789)
0619 13:00:53.991919 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:53.991923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:53.993026 (+ 1103us) server_negotiation.cc:407] Connection header received
0619 13:00:53.994720 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:53.994724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:53.994784 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:53.994870 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:53.996284 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:53.996957 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:53.998532 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:53.999161 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:53.999256 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.003287 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:54.003296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:54.003304 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:54.003380 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:54.006935 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.007932 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.007936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.007938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.008053 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.008608 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.008612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.008614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.008875 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:54.008996 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:54.009886 (+ 890us) server_negotiation.cc:299] Negotiation successful
0619 13:00:54.010056 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:00:54.011873 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:53.992188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:43789 (local address 127.1.87.66:49283)
0619 13:00:53.992868 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:53.992904 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:53.994476 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:53.995063 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:53.995071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:53.995623 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:53.996143 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:53.996159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:53.997077 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:53.997081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:53.998398 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:53.998410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:53.999410 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:53.999424 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:53.999624 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.000390 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:54.000424 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:54.003049 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:54.007102 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:54.007116 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:54.007137 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:54.007812 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:54.008168 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:54.008176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:54.008182 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:54.008427 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:54.009005 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:54.009017 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:54.009494 (+ 477us) client_negotiation.cc:769] Sending connection context
0619 13:00:54.009786 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 13:00:54.010387 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":278,"threads_started":1}
I20260619 13:00:54.013406 7332 heartbeater.cc:347] Connected to a master server at 127.1.87.126:43789
I20260619 13:00:54.013724 7332 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:54.014545 7332 heartbeater.cc:511] Master 127.1.87.126:43789 requested a full tablet report, sending...
I20260619 13:00:54.016077 7009 ts_manager.cc:194] Registered new tserver with Master: e8bb64e5c70f4156ae503a277577fa13 (127.1.87.66:38379)
I20260619 13:00:54.017344 7009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:49283
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:00:54.045666 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43789
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:54.362934 7337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:54.363243 7337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:54.363436 7337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:54.370494 7337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:54.370656 7337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:54.370841 7337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:54.380326 7337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:54.381573 7337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:54.382848 7337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:54.395248 7343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:54.395305 7345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:54.395345 7342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:54.396081 7337 server_base.cc:1034] running on GCE node
I20260619 13:00:54.396637 7337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:54.397466 7337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:54.398700 7337 hybrid_clock.cc:648] HybridClock initialized: now 1781874054398655 us; error 52 us; skew 500 ppm
I20260619 13:00:54.398998 7337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:54.401901 7337 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:00:54.403177 7337 webserver.cc:460] Webserver started at http://127.1.87.67:41987/ using document root <none> and password file <none>
I20260619 13:00:54.403853 7337 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:54.403993 7337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:54.404285 7337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:54.406901 7337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "1f4125f48d0a407a93726b80ce6a60ab"
format_stamp: "Formatted at 2026-06-19 13:00:54 on dist-test-slave-tpsm"
server_key: "047d8d1da065d54f921c7f79e2d5dce5"
server_key_iv: "02487f08573515b97618e6b834e6c409"
server_key_version: "encryptionkey@0"
I20260619 13:00:54.407673 7337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "1f4125f48d0a407a93726b80ce6a60ab"
format_stamp: "Formatted at 2026-06-19 13:00:54 on dist-test-slave-tpsm"
server_key: "047d8d1da065d54f921c7f79e2d5dce5"
server_key_iv: "02487f08573515b97618e6b834e6c409"
server_key_version: "encryptionkey@0"
I20260619 13:00:54.412292 7337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 13:00:54.415481 7352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:54.416699 7337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:00:54.416874 7337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "1f4125f48d0a407a93726b80ce6a60ab"
format_stamp: "Formatted at 2026-06-19 13:00:54 on dist-test-slave-tpsm"
server_key: "047d8d1da065d54f921c7f79e2d5dce5"
server_key_iv: "02487f08573515b97618e6b834e6c409"
server_key_version: "encryptionkey@0"
I20260619 13:00:54.417066 7337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:54.436352 7337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:54.440048 7337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:54.440330 7337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:54.441226 7337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:54.442478 7337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:54.442580 7337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:54.442739 7337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:54.442811 7337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:54.479120 7337 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:38843
I20260619 13:00:54.479153 7464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:38843 every 8 connection(s)
I20260619 13:00:54.480387 7337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:54.484499 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 7337
I20260619 13:00:54.484681 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:54.485203 1373 external_mini_cluster.cc:1428] Setting key 2e57a7378a4fff65b8365553c8fff6cf
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:00:54.500047 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:54.484215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:38947 (local address 127.1.87.126:43789)
0619 13:00:54.485853 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.485859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.485881 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:00:54.486935 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.486940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.487028 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.487136 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.488662 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.489406 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.490900 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.491295 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.491475 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.494639 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:54.494644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:54.494647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:54.494675 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:54.497030 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.497670 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.497673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.497675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.497790 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.498147 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.498150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.498152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.498453 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:54.498605 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:54.499457 (+ 852us) server_negotiation.cc:299] Negotiation successful
0619 13:00:54.499674 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1510}
I20260619 13:00:54.500943 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:54.483700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:43789 (local address 127.1.87.67:38947)
0619 13:00:54.484624 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:54.484681 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:54.486636 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:54.487421 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:54.487429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:54.487940 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:54.488505 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:54.488524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.489558 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.489570 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:54.490763 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:54.490774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.491556 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.491566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:54.491795 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.492401 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:54.492426 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:54.494467 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:54.497150 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:54.497156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:54.497167 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:54.497560 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:54.497880 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:54.497883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:54.497885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:54.498035 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:54.498604 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:54.498611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:54.499006 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 13:00:54.499353 (+ 347us) client_negotiation.cc:240] Negotiation successful
0619 13:00:54.499801 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":473,"threads_started":1}
I20260619 13:00:54.502231 7465 heartbeater.cc:347] Connected to a master server at 127.1.87.126:43789
I20260619 13:00:54.502521 7465 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:54.503206 7465 heartbeater.cc:511] Master 127.1.87.126:43789 requested a full tablet report, sending...
I20260619 13:00:54.504539 7009 ts_manager.cc:194] Registered new tserver with Master: 1f4125f48d0a407a93726b80ce6a60ab (127.1.87.67:38843)
I20260619 13:00:54.505316 7009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:38947
I20260619 13:00:54.512408 7198 heartbeater.cc:503] Master 127.1.87.126:43789 was elected leader, sending a full tablet report...
I20260619 13:00:54.512681 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/test-admin.keytab.
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:54.538018 1373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 13:00:54.543490 7472 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:00:54.540804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.1.87.65:34773)
0619 13:00:54.541444 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.541455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.541489 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:00:54.541574 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.541581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.541791 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.541967 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.543022 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33090: BlockingRecv error: recv got EOF from 127.0.0.1:33090 (error 108)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":279,"threads_started":1}
W20260619 13:00:54.543665 1373 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5cc' of type FILE
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:54.545253 1373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:54.546664 1373 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:54.562137 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:54.548508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.1.87.65:34773)
0619 13:00:54.548749 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.548755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.548990 (+ 235us) server_negotiation.cc:407] Connection header received
0619 13:00:54.549181 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.549186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.549266 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.549377 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.550431 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.551520 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.552748 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.553319 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.553425 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.557050 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:54.557062 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:54.557077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:54.557111 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:54.559841 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.560457 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.560464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.560467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.560576 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.560879 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.560882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.560884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.561217 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:54.561447 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:54.561684 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 13:00:54.561867 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260619 13:00:54.567655 7472 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:00:54.566052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.1.87.65:34773)
0619 13:00:54.566223 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.566228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.566552 (+ 324us) server_negotiation.cc:407] Connection header received
0619 13:00:54.566685 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.566688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.566749 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.566825 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.567428 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33118: BlockingRecv error: recv got EOF from 127.0.0.1:33118 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 13:00:54.567866 1373 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5cc' of type FILE
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:54.569917 1373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:54.571239 1373 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:54.585732 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:54.572764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.1.87.65:34773)
0619 13:00:54.572949 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.572956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.573029 (+ 73us) server_negotiation.cc:407] Connection header received
0619 13:00:54.573207 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.573219 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.573271 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.573379 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.574276 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.574928 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.576216 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.576624 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.576805 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.580506 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:54.580512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:54.580517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:54.580571 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:54.582949 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.583711 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.583718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.583722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.583848 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.584213 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.584218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.584221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.584692 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:54.584986 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:54.585247 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 13:00:54.585442 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 13:00:54.590016 7472 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:00:54.588325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.1.87.65:34773)
0619 13:00:54.588481 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.588489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.588807 (+ 318us) server_negotiation.cc:407] Connection header received
0619 13:00:54.588981 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.588986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.589062 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.589158 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.589797 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33142: BlockingRecv error: recv got EOF from 127.0.0.1:33142 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 13:00:54.590440 1373 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874004469744-1373-0/krb5kdc/krb5cc)
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:54.591948 1373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:54.593226 1373 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:00:54.606997 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:54.594553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.1.87.65:34773)
0619 13:00:54.594745 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.594750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.594867 (+ 117us) server_negotiation.cc:407] Connection header received
0619 13:00:54.595036 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.595042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.595104 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.595199 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.596179 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.596870 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.598042 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.598453 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.598573 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.602138 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:54.602142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:54.602146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:54.602175 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:54.604558 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.605147 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.605153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.605157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.605280 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.605648 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.605654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.605658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.605933 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:54.606236 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:54.606483 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 13:00:54.606680 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:00:54.608554 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 7070
I20260619 13:00:54.619550 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:54.609226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.1.87.65:34773)
0619 13:00:54.609391 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.609395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.609776 (+ 381us) server_negotiation.cc:407] Connection header received
0619 13:00:54.609898 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.609902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.609954 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.610027 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.610857 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.611594 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.612707 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.613075 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.613175 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.615107 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:54.615112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:54.615115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:54.615142 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:54.617426 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.617975 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.617979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.617981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.618105 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.618400 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.618403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.618405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.618629 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:54.618845 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:54.619052 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 13:00:54.619226 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:00:54.620070 7193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:54.705636 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 7070
I20260619 13:00:54.721940 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 7204
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:00:54.738943 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:54.722901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.1.87.66:38379)
0619 13:00:54.723497 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.723504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.723529 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:00:54.723588 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.723594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.723800 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.723928 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.725027 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.726309 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.727672 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.728258 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.728378 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.732888 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:54.732899 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:54.732912 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:54.732946 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:54.736171 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.736968 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.736975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.736978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.737098 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.737472 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.737476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.737478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.737838 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:54.738070 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:54.738352 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 13:00:54.738650 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":302,"threads_started":1}
I20260619 13:00:54.740295 7327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:54.822090 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 7204
I20260619 13:00:54.838063 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 7337
Jun 19 13:00:54 dist-test-slave-tpsm krb5kdc[6964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:00:54.853420 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:54.838940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.1.87.67:38843)
0619 13:00:54.839586 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:54.839593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:54.839622 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:00:54.839680 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:54.839687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:54.839893 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:54.840018 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:54.840967 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.841951 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.843083 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:54.843764 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:54.843859 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:54.847857 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:54.847869 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:54.847883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:54.847914 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:54.850665 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.851362 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.851374 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.851380 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.851517 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:54.851896 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:54.851903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:54.851907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:54.852259 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:54.852567 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:54.852872 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 13:00:54.853134 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":318,"threads_started":1}
I20260619 13:00:54.854604 7460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:54.936471 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 7337
I20260619 13:00:54.956261 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 6977
I20260619 13:00:54.957401 7039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:55.057327 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 6977
2026-06-19T13:00:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3922 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:00:55 dist-test-slave-tpsm krb5kdc[7492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:00:55 dist-test-slave-tpsm krb5kdc[7492](info): set up 2 sockets
Jun 19 13:00:55 dist-test-slave-tpsm krb5kdc[7492](info): commencing operation
krb5kdc: starting...
W20260619 13:00:56.152043 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.011s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:00:56 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:00:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.126.keytab.
Entry for principal oryx/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.126.keytab.
I20260619 13:00:56.355948 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:33027
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.126.keytab
--principal=oryx/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:33027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:56.673329 7505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:56.673616 7505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:56.673765 7505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:56.680994 7505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:56.681109 7505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:56.681190 7505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:56.681300 7505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:56.681362 7505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:56.691263 7505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:33027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.126.keytab
--principal=oryx/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:33027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:56.692618 7505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:56.693867 7505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:56.710992 7511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:56.711083 7510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:56.711040 7513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:56.712335 7505 server_base.cc:1034] running on GCE node
I20260619 13:00:56.713119 7505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:56.714512 7505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:56.715746 7505 hybrid_clock.cc:648] HybridClock initialized: now 1781874056715703 us; error 64 us; skew 500 ppm
I20260619 13:00:56.716101 7505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:56 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:56.721884 7505 init.cc:377] Logged in from keytab as oryx/127.1.87.126@KRBTEST.COM (short username oryx)
I20260619 13:00:56.723363 7505 webserver.cc:460] Webserver started at http://127.1.87.126:42431/ using document root <none> and password file <none>
I20260619 13:00:56.724006 7505 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:56.724113 7505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:56.724342 7505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:56.727254 7505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "9933c1c7f5644e159c996c2efbaabcb6"
format_stamp: "Formatted at 2026-06-19 13:00:56 on dist-test-slave-tpsm"
server_key: "58b226ebbab7d93927dff807e120463e"
server_key_iv: "e88f929656d0d5090ed2d17fe585e888"
server_key_version: "encryptionkey@0"
I20260619 13:00:56.728045 7505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "9933c1c7f5644e159c996c2efbaabcb6"
format_stamp: "Formatted at 2026-06-19 13:00:56 on dist-test-slave-tpsm"
server_key: "58b226ebbab7d93927dff807e120463e"
server_key_iv: "e88f929656d0d5090ed2d17fe585e888"
server_key_version: "encryptionkey@0"
I20260619 13:00:56.733068 7505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 13:00:56.736637 7520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:56.738027 7505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:00:56.738222 7505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "9933c1c7f5644e159c996c2efbaabcb6"
format_stamp: "Formatted at 2026-06-19 13:00:56 on dist-test-slave-tpsm"
server_key: "58b226ebbab7d93927dff807e120463e"
server_key_iv: "e88f929656d0d5090ed2d17fe585e888"
server_key_version: "encryptionkey@0"
I20260619 13:00:56.738410 7505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:56.777371 7505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:56.783139 7505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:56.783564 7505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:56.803844 7505 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:33027
I20260619 13:00:56.803879 7571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:33027 every 8 connection(s)
I20260619 13:00:56.805198 7505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:00:56.806923 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 7505
I20260619 13:00:56.807125 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:00:56.807840 1373 external_mini_cluster.cc:1428] Setting key 72980cc1909df3130df5d22dcb0a6c14
I20260619 13:00:56.809324 7572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:00:56 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:00:56.824031 7572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: Bootstrap starting.
I20260619 13:00:56.827152 7572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:56.828269 7572 log.cc:824] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:56.830818 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:56.809905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.1.87.126:33027)
0619 13:00:56.810765 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:56.810776 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:56.810817 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:00:56.812364 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:56.812384 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:56.812780 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:56.813103 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:56.815012 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:56.816473 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:56.818223 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:56.818906 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:56.819034 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:56.823715 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:56.823724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:56.823740 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:56.823780 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:56.826773 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:56.827589 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:56.827596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:56.827603 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:56.827765 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:56.828135 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:56.828140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:56.828145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:56.828732 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:56.829071 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:56.829461 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 13:00:56.829866 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":377,"threads_started":1}
I20260619 13:00:56.831293 7572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: No bootstrap required, opened a new log
I20260619 13:00:56.840054 7572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } }
I20260619 13:00:56.840476 7572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:56.840586 7572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9933c1c7f5644e159c996c2efbaabcb6, State: Initialized, Role: FOLLOWER
I20260619 13:00:56.841152 7572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } }
I20260619 13:00:56.841398 7572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:00:56.841564 7572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:00:56.841741 7572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:56.844082 7572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } }
I20260619 13:00:56.844594 7572 leader_election.cc:302] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9933c1c7f5644e159c996c2efbaabcb6; no voters:
I20260619 13:00:56.845258 7572 leader_election.cc:288] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:00:56.845410 7577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:00:56.846177 7577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 9933c1c7f5644e159c996c2efbaabcb6, State: Running, Role: LEADER
I20260619 13:00:56.846679 7577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } }
I20260619 13:00:56.847507 7572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:56.849700 7579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9933c1c7f5644e159c996c2efbaabcb6. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } }
I20260619 13:00:56.850047 7579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:56.849802 7578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } }
I20260619 13:00:56.850189 7578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:56.850972 7584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:00:56.856559 7584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:00:56.864938 7584 catalog_manager.cc:1347] Generated new cluster ID: 5cd6dc33edf249409d8ef737f361d0a8
I20260619 13:00:56.865062 7584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:00:56.891391 7584 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:00:56.893325 7584 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:00:56.903529 7584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: Generated new TSK 0
I20260619 13:00:56.904521 7584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab.
Entry for principal oryx/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab.
I20260619 13:00:56.956595 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33027
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab
--principal=oryx/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:57.276638 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:57.276911 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:57.277001 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:57.284101 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:57.284256 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:57.284399 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:00:57.294045 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab
--principal=oryx/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:57.295359 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:57.296551 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:57.309005 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:57.309038 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:57.310127 7598 server_base.cc:1034] running on GCE node
W20260619 13:00:57.308959 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:57.311033 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:57.311980 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:57.313184 7598 hybrid_clock.cc:648] HybridClock initialized: now 1781874057313131 us; error 45 us; skew 500 ppm
I20260619 13:00:57.313503 7598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:57 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874057, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:57.316720 7598 init.cc:377] Logged in from keytab as oryx/127.1.87.65@KRBTEST.COM (short username oryx)
I20260619 13:00:57.318126 7598 webserver.cc:460] Webserver started at http://127.1.87.65:45365/ using document root <none> and password file <none>
I20260619 13:00:57.318809 7598 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:57.318950 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:57.319243 7598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:57.322129 7598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "a1b9fc73b1714ddabd5d5cd927adacf1"
format_stamp: "Formatted at 2026-06-19 13:00:57 on dist-test-slave-tpsm"
server_key: "b7212c6f6f59a3e24da8543242b47967"
server_key_iv: "7f68ea78b5b3414f7da9990f352b43cd"
server_key_version: "encryptionkey@0"
I20260619 13:00:57.322861 7598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "a1b9fc73b1714ddabd5d5cd927adacf1"
format_stamp: "Formatted at 2026-06-19 13:00:57 on dist-test-slave-tpsm"
server_key: "b7212c6f6f59a3e24da8543242b47967"
server_key_iv: "7f68ea78b5b3414f7da9990f352b43cd"
server_key_version: "encryptionkey@0"
I20260619 13:00:57.327850 7598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:00:57.331248 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:57.332722 7598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:00:57.332896 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "a1b9fc73b1714ddabd5d5cd927adacf1"
format_stamp: "Formatted at 2026-06-19 13:00:57 on dist-test-slave-tpsm"
server_key: "b7212c6f6f59a3e24da8543242b47967"
server_key_iv: "7f68ea78b5b3414f7da9990f352b43cd"
server_key_version: "encryptionkey@0"
I20260619 13:00:57.333099 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:57.360291 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:57.364603 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:57.364910 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:57.365887 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:57.367187 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:57.367287 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:57.367462 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:57.367511 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:57.400089 7598 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:42863
I20260619 13:00:57.400187 7725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:42863 every 8 connection(s)
I20260619 13:00:57.401311 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:00:57.406678 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 7598
I20260619 13:00:57.406886 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:00:57.407716 1373 external_mini_cluster.cc:1428] Setting key 9d0b0645457389c867827e18689e534d
Jun 19 13:00:57 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874057, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:00:57.424261 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:57.404109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:49853 (local address 127.1.87.126:33027)
0619 13:00:57.404535 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:57.404542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:57.405452 (+ 910us) server_negotiation.cc:407] Connection header received
0619 13:00:57.407885 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:57.407889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:57.407952 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:57.408045 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:57.409790 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:57.410456 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:57.412975 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:57.413387 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:57.413489 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:57.417961 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:57.417966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:57.417969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:57.418000 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:57.420619 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:57.421576 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:57.421581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:57.421583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:57.421713 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:57.422213 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:57.422216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:57.422219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:57.422483 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:57.422599 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:57.423827 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0619 13:00:57.423990 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":208,"threads_started":1}
I20260619 13:00:57.425918 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:57.404599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.65:49853)
0619 13:00:57.405294 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:57.405335 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:57.407573 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:57.408310 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:57.408323 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:57.408902 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:57.409604 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:57.409622 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:57.410618 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:57.410627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:57.412809 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:57.412826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:57.413619 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:57.413628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:57.413949 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:57.414691 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:57.414726 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:57.417718 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:57.420761 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:57.420772 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:57.420792 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:57.421420 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:57.421828 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:57.421835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:57.421841 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:57.422070 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:57.422753 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:57.422768 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:57.423389 (+ 621us) client_negotiation.cc:769] Sending connection context
0619 13:00:57.423700 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 13:00:57.424290 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":276,"threads_started":1}
I20260619 13:00:57.427488 7726 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33027
I20260619 13:00:57.427802 7726 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:57.428485 7726 heartbeater.cc:511] Master 127.1.87.126:33027 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.1.87.66@KRBTEST.COM; defaulting to no policy
I20260619 13:00:57.430461 7537 ts_manager.cc:194] Registered new tserver with Master: a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863)
I20260619 13:00:57.432096 7537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:49853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab.
Entry for principal oryx/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab.
I20260619 13:00:57.454655 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33027
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab
--principal=oryx/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:57.767652 7732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:57.767922 7732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:57.768102 7732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:57.775614 7732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:57.775775 7732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:57.775980 7732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:00:57.785560 7732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab
--principal=oryx/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:57.786870 7732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:57.788179 7732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:57.801010 7737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:57.800938 7738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:57.800977 7740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:57.801712 7732 server_base.cc:1034] running on GCE node
I20260619 13:00:57.802310 7732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:57.803160 7732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:57.804384 7732 hybrid_clock.cc:648] HybridClock initialized: now 1781874057804327 us; error 50 us; skew 500 ppm
I20260619 13:00:57.804690 7732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:57 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874057, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:57.807701 7732 init.cc:377] Logged in from keytab as oryx/127.1.87.66@KRBTEST.COM (short username oryx)
I20260619 13:00:57.809226 7732 webserver.cc:460] Webserver started at http://127.1.87.66:41613/ using document root <none> and password file <none>
I20260619 13:00:57.809897 7732 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:57.810016 7732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:57.810319 7732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:57.813110 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "259e01b38a444737934f88d0c1ccad01"
format_stamp: "Formatted at 2026-06-19 13:00:57 on dist-test-slave-tpsm"
server_key: "c248764194da5348bfe84219f0a341b1"
server_key_iv: "402f447fe0381166507d6029e960f9a3"
server_key_version: "encryptionkey@0"
I20260619 13:00:57.813834 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "259e01b38a444737934f88d0c1ccad01"
format_stamp: "Formatted at 2026-06-19 13:00:57 on dist-test-slave-tpsm"
server_key: "c248764194da5348bfe84219f0a341b1"
server_key_iv: "402f447fe0381166507d6029e960f9a3"
server_key_version: "encryptionkey@0"
I20260619 13:00:57.818610 7732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:00:57.821926 7747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:57.823167 7732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:00:57.823355 7732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "259e01b38a444737934f88d0c1ccad01"
format_stamp: "Formatted at 2026-06-19 13:00:57 on dist-test-slave-tpsm"
server_key: "c248764194da5348bfe84219f0a341b1"
server_key_iv: "402f447fe0381166507d6029e960f9a3"
server_key_version: "encryptionkey@0"
I20260619 13:00:57.823563 7732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:57.838438 7732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:57.842723 7732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:57.843004 7732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:57.844081 7732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:57.845315 7732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:57.845387 7732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:57.845547 7732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:57.845600 7732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:57.878602 7732 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:43129
I20260619 13:00:57.878671 7859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:43129 every 8 connection(s)
I20260619 13:00:57.879877 7732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:00:57.883387 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 7732
I20260619 13:00:57.883564 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:00:57.884070 1373 external_mini_cluster.cc:1428] Setting key e8625c6bbef0796295c26833da896b9b
Jun 19 13:00:57 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874057, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:00:57.900487 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:57.882326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:38343 (local address 127.1.87.126:33027)
0619 13:00:57.882552 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:57.882557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:57.883840 (+ 1283us) server_negotiation.cc:407] Connection header received
0619 13:00:57.886015 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:57.886021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:57.886110 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:57.886242 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:57.887827 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:57.888568 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:57.890195 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:57.890642 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:57.890788 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:57.894414 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:57.894429 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:57.894433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:57.894466 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:57.897206 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:57.897989 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:57.897995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:57.898011 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:57.898150 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:57.898623 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:57.898634 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:57.898636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:57.898897 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:57.899066 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:57.899849 (+ 783us) server_negotiation.cc:299] Negotiation successful
0619 13:00:57.900088 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 13:00:57.901477 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:57.882738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.66:38343)
0619 13:00:57.883666 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:57.883727 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:57.885649 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:57.886482 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:57.886490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:57.886997 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:57.887649 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:57.887668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:57.888739 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:57.888752 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:57.890036 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:57.890049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:57.890881 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:57.890889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:57.891147 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:57.891934 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:57.891958 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:57.894186 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:57.897371 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:57.897381 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:57.897400 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:57.897849 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:57.898276 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:57.898281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:57.898285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:57.898483 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:57.899046 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:57.899055 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:57.899475 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 13:00:57.899765 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 13:00:57.900307 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":358,"threads_started":1}
WARNING: no policy specified for oryx/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:00:57.903126 7860 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33027
I20260619 13:00:57.903537 7860 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:57.904430 7860 heartbeater.cc:511] Master 127.1.87.126:33027 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.67@KRBTEST.COM" created.
I20260619 13:00:57.906085 7537 ts_manager.cc:194] Registered new tserver with Master: 259e01b38a444737934f88d0c1ccad01 (127.1.87.66:43129)
I20260619 13:00:57.906859 7537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:38343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab.
Entry for principal oryx/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab.
I20260619 13:00:57.928169 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33027
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab
--principal=oryx/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:00:58.240319 7865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:58.240566 7865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:58.240661 7865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:58.247841 7865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:58.247936 7865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:58.248040 7865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:00:58.257360 7865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab
--principal=oryx/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:58.258497 7865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:58.259826 7865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:58.272583 7871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:58.272701 7870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:58.272696 7873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:58.273460 7865 server_base.cc:1034] running on GCE node
I20260619 13:00:58.274088 7865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:58.275020 7865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:58.276253 7865 hybrid_clock.cc:648] HybridClock initialized: now 1781874058276174 us; error 73 us; skew 500 ppm
I20260619 13:00:58.276568 7865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:58 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874058, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:58.279700 7865 init.cc:377] Logged in from keytab as oryx/127.1.87.67@KRBTEST.COM (short username oryx)
I20260619 13:00:58.281100 7865 webserver.cc:460] Webserver started at http://127.1.87.67:35793/ using document root <none> and password file <none>
I20260619 13:00:58.281767 7865 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:58.281909 7865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:58.282202 7865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:58.284979 7865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "f1707fe2ccdf4c9fbb06aa52a092446b"
format_stamp: "Formatted at 2026-06-19 13:00:58 on dist-test-slave-tpsm"
server_key: "3f5ea1532211d1c6c1181300b5b0e796"
server_key_iv: "04a76b0c62ab1799a3aa555c4fadfc70"
server_key_version: "encryptionkey@0"
I20260619 13:00:58.285717 7865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "f1707fe2ccdf4c9fbb06aa52a092446b"
format_stamp: "Formatted at 2026-06-19 13:00:58 on dist-test-slave-tpsm"
server_key: "3f5ea1532211d1c6c1181300b5b0e796"
server_key_iv: "04a76b0c62ab1799a3aa555c4fadfc70"
server_key_version: "encryptionkey@0"
I20260619 13:00:58.290917 7865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:00:58.294165 7880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:58.295468 7865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:00:58.295640 7865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "f1707fe2ccdf4c9fbb06aa52a092446b"
format_stamp: "Formatted at 2026-06-19 13:00:58 on dist-test-slave-tpsm"
server_key: "3f5ea1532211d1c6c1181300b5b0e796"
server_key_iv: "04a76b0c62ab1799a3aa555c4fadfc70"
server_key_version: "encryptionkey@0"
I20260619 13:00:58.295840 7865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:58.315200 7865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:58.319099 7865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:58.319450 7865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:58.320392 7865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:00:58.321743 7865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:00:58.321847 7865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:58.321995 7865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:00:58.322073 7865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:58.357070 7865 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:36091
I20260619 13:00:58.357098 7992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:36091 every 8 connection(s)
I20260619 13:00:58.358381 7865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:00:58.367853 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 7865
I20260619 13:00:58.368028 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:00:58.368481 1373 external_mini_cluster.cc:1428] Setting key 15748b79083bfbeceb32392a9f9acdbc
Jun 19 13:00:58 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874058, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:00:58.377310 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.360857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:51229 (local address 127.1.87.126:33027)
0619 13:00:58.361017 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:58.361022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:58.362108 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 13:00:58.364011 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:58.364017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:58.364133 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:58.364249 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:58.365631 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.366340 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.368124 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.368603 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.368711 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.372505 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:58.372510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:58.372513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:58.372540 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:58.374795 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.375517 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.375522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.375524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.375630 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.376023 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.376026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.376027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.376261 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:58.376369 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:58.376936 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 13:00:58.377075 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:00:58.378199 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.361245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.67:51229)
0619 13:00:58.361942 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:58.361977 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:58.363753 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:58.364476 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:58.364484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:58.364951 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:58.365461 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.365476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.366488 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.366498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.367960 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.367976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.368826 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.368834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.369101 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.369893 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:58.369928 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:58.372333 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:58.374925 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.374930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.374942 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.375415 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.375741 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.375745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.375747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.375925 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.376364 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:58.376369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:58.376652 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 13:00:58.376861 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 13:00:58.377197 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":293,"threads_started":1}
I20260619 13:00:58.379463 7993 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33027
I20260619 13:00:58.379781 7993 heartbeater.cc:464] Registering TS with master...
I20260619 13:00:58.380465 7993 heartbeater.cc:511] Master 127.1.87.126:33027 requested a full tablet report, sending...
I20260619 13:00:58.381912 7537 ts_manager.cc:194] Registered new tserver with Master: f1707fe2ccdf4c9fbb06aa52a092446b (127.1.87.67:36091)
I20260619 13:00:58.382854 7537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:51229
I20260619 13:00:58.383626 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.87.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.125.keytab.
Entry for principal oryx/127.1.87.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.125.keytab.
I20260619 13:00:58.428197 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.125:43867
--webserver_interface=127.1.87.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.125.keytab
--principal=oryx/127.1.87.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:33027,127.1.87.125:43867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:00:58.434787 7726 heartbeater.cc:503] Master 127.1.87.126:33027 was elected leader, sending a full tablet report...
W20260619 13:00:58.742522 7998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:58.742842 7998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:58.743005 7998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:00:58.750300 7998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:58.750445 7998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:58.750555 7998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:58.750660 7998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:58.750751 7998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:58.760586 7998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:33027,127.1.87.125:43867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.125.keytab
--principal=oryx/127.1.87.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.125:43867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.87.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
W20260619 13:00:58.761015 7998 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.87.126:33027,127.1.87.125:43867'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 13:00:58.762038 7998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:58.763345 7998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:58.774580 8003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:58.774726 8004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:58.774699 8006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:58.775864 7998 server_base.cc:1034] running on GCE node
I20260619 13:00:58.776717 7998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:58.778115 7998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:58.779414 7998 hybrid_clock.cc:648] HybridClock initialized: now 1781874058779274 us; error 35 us; skew 500 ppm
I20260619 13:00:58.779783 7998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:58 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874058, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:58.783125 7998 init.cc:377] Logged in from keytab as oryx/127.1.87.125@KRBTEST.COM (short username oryx)
I20260619 13:00:58.784680 7998 webserver.cc:460] Webserver started at http://127.1.87.125:40155/ using document root <none> and password file <none>
I20260619 13:00:58.785375 7998 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:58.785462 7998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:58.785761 7998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:58.788753 7998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data/instance:
uuid: "14031fd252a44b91b756b343b29303c9"
format_stamp: "Formatted at 2026-06-19 13:00:58 on dist-test-slave-tpsm"
server_key: "ca807f887f0090cad0f4d9c94a7fb447"
server_key_iv: "58394997a7e6a199c0be44d2204c5b4d"
server_key_version: "encryptionkey@0"
I20260619 13:00:58.789516 7998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal/instance:
uuid: "14031fd252a44b91b756b343b29303c9"
format_stamp: "Formatted at 2026-06-19 13:00:58 on dist-test-slave-tpsm"
server_key: "ca807f887f0090cad0f4d9c94a7fb447"
server_key_iv: "58394997a7e6a199c0be44d2204c5b4d"
server_key_version: "encryptionkey@0"
I20260619 13:00:58.794386 7998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 13:00:58.797891 8013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:58.799281 7998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:00:58.799517 7998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal
uuid: "14031fd252a44b91b756b343b29303c9"
format_stamp: "Formatted at 2026-06-19 13:00:58 on dist-test-slave-tpsm"
server_key: "ca807f887f0090cad0f4d9c94a7fb447"
server_key_iv: "58394997a7e6a199c0be44d2204c5b4d"
server_key_version: "encryptionkey@0"
I20260619 13:00:58.799738 7998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:58.811913 7998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:58.815678 7998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:58.815927 7998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:58.833844 7998 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.125:43867
I20260619 13:00:58.833915 8064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.125:43867 every 8 connection(s)
I20260619 13:00:58.835175 7998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
I20260619 13:00:58.838990 8065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:58.844326 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 7998
I20260619 13:00:58.846050 8065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:00:58 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874058, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.125@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
Jun 19 13:00:58 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.125@KRBTEST.COM
I20260619 13:00:58.864154 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.848828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.1.87.126:33027)
0619 13:00:58.849035 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:58.849041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:58.849686 (+ 645us) server_negotiation.cc:407] Connection header received
0619 13:00:58.850082 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:58.850087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:58.850170 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:58.850278 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:58.851496 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.852313 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.853826 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.854217 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.854347 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.857728 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:58.857734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:58.857739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:58.857770 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:58.860685 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.861617 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.861620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.861623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.861745 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.862245 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.862250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.862254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.862672 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:58.862807 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:58.863600 (+ 793us) server_negotiation.cc:299] Negotiation successful
0619 13:00:58.863792 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:00:58.865322 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.846480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.1.87.125:43867)
0619 13:00:58.847258 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:58.847269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:58.847306 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:00:58.849705 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:58.849727 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:58.849999 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:58.850200 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:58.851619 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.852778 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.853927 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.854439 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.854678 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.858619 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:58.858626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:58.858636 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:58.858668 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:58.861904 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.862708 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.862713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.862715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.862869 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.863210 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.863218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.863220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.863642 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:58.863923 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:58.864371 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 13:00:58.864596 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":552,"thread_start_us":354,"threads_started":1}
I20260619 13:00:58.865362 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.848857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:33646)
0619 13:00:58.849543 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:58.849585 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:58.849874 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:58.850339 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:58.850345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:58.850937 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:58.851284 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.851298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.852452 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.852457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.853681 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.853692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.854437 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.854444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.854662 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.855296 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:58.855370 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:58.857490 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:58.860812 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.860821 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.860832 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.861491 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.861855 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.861860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.861863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.862113 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.862797 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:58.862806 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:58.863183 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 13:00:58.863527 (+ 344us) client_negotiation.cc:240] Negotiation successful
0619 13:00:58.863988 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":38,"spinlock_wait_cycles":19712,"thread_start_us":357,"threads_started":1}
I20260619 13:00:58.867110 8065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:00:58 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874058, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.125@KRBTEST.COM for oryx/127.1.87.125@KRBTEST.COM
I20260619 13:00:58.878496 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.867756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.0.0.1:60332)
0619 13:00:58.867898 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:58.867916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:58.868003 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:58.868305 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:58.868308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:58.868580 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:58.868698 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.868705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.869579 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.869582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.870439 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.870449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.871073 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.871077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.871224 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.871854 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:58.871870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:58.873692 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:58.876458 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.876462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.876464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.876783 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.877073 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.877076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.877078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.877204 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.877627 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:58.877630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:58.877734 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 13:00:58.877893 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 13:00:58.878166 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 13:00:58.878533 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.867805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.1.87.125:43867)
0619 13:00:58.867958 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:58.867966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:58.867984 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:00:58.868115 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:58.868120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:58.868187 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:58.868258 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:58.868806 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.869456 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.870553 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.870961 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.871053 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.873850 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:58.873855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:58.873858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:58.873884 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:58.876345 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.876876 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.876880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.876882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.876986 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.877289 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.877291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.877293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.877536 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:58.877636 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:58.877963 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 13:00:58.878321 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":31}
I20260619 13:00:58.884246 8065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: Bootstrap starting.
I20260619 13:00:58.887049 8065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:58.888031 8065 log.cc:824] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:58.890311 8065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: No bootstrap required, opened a new log
I20260619 13:00:58.896904 8065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } }
I20260619 13:00:58.897195 8065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:58.897269 8065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14031fd252a44b91b756b343b29303c9, State: Initialized, Role: FOLLOWER
I20260619 13:00:58.897727 8065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } }
I20260619 13:00:58.898447 8071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } } }
I20260619 13:00:58.898777 8071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:00:58.899593 8065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:00:58.907793 8082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:00:58.907922 8082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:00:58.909370 7860 heartbeater.cc:503] Master 127.1.87.126:33027 was elected leader, sending a full tablet report...
I20260619 13:00:58.910447 7998 master_runner.cc:428] Detected that this master 14031fd252a44b91b756b343b29303c9 is joining an existing cluster
I20260619 13:00:58.910537 7998 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.87.125:43867
I20260619 13:00:58.931965 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.914449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:33658)
0619 13:00:58.915116 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:58.915135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:58.915278 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:58.915927 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:58.915934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:58.916851 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:58.917326 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.917342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.918785 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.918790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.920544 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.920557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.921849 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.921858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.922079 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.922785 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:58.922814 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:58.924054 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:58.928690 (+ 4636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.928696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.928702 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.929228 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.929630 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.929637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.929644 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.929823 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.930544 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:58.930552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:58.930771 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 13:00:58.930937 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 13:00:58.931306 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":1,"thread_start_us":386,"threads_started":1}
I20260619 13:00:58.932951 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.915384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.1.87.125:43867)
0619 13:00:58.915533 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:58.915539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:58.918002 (+ 2463us) server_negotiation.cc:407] Connection header received
0619 13:00:58.918183 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:58.918188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:58.918246 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:58.918341 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:58.919717 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.921255 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.923156 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.923959 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.924050 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.926369 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:58.926380 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:58.926388 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:58.926424 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:58.929991 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.930778 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.930784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.930789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.930931 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.931366 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.931370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.931374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.931734 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:58.931865 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:58.932364 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 13:00:58.932582 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":372,"server-negotiator.queue_time_us":54}
I20260619 13:00:58.933109 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.914386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.1.87.126:33027)
0619 13:00:58.914578 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:58.914585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:58.915233 (+ 648us) server_negotiation.cc:407] Connection header received
0619 13:00:58.915454 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:58.915459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:58.915548 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:58.915640 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:58.917503 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.918616 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.920694 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.921440 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.921582 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.924281 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:58.924289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:58.924297 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:58.924341 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:58.928529 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.929379 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.929384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.929388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.929520 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.929954 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.929961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.929968 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.930343 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:58.930482 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:58.932436 (+ 1954us) server_negotiation.cc:299] Negotiation successful
0619 13:00:58.932672 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:00:58.933868 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.915405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.0.0.1:60342)
0619 13:00:58.917908 (+ 2503us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:58.917923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:58.918049 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:58.918481 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:58.918484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:58.919198 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:58.919584 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.919596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.921443 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.921448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.922998 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.923012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.924333 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.924339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.924493 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.925075 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:58.925106 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:58.926191 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:58.930135 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.930140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.930144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.930647 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.931067 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.931072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.931075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.931230 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.931972 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:58.931978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:58.932135 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 13:00:58.933173 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0619 13:00:58.933454 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2398,"thread_start_us":2213,"threads_started":1}
WARNING: no policy specified for oryx/127.1.87.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.124@KRBTEST.COM" created.
Jun 19 13:00:58 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.126@KRBTEST.COM for oryx/127.1.87.125@KRBTEST.COM
I20260619 13:00:58.961352 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.940814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.0.0.1:60356)
0619 13:00:58.941648 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:58.941694 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:58.941953 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:58.942526 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:58.942534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:58.943447 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:58.943712 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.943725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.945225 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.945231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.946789 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:58.946804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.947750 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.947756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:58.947955 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.948730 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:58.948747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:58.950796 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:58.954034 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.954047 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.954058 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.954645 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.955081 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:58.955087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:58.955092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:58.955261 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:58.960065 (+ 4804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:58.960076 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:58.960314 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 13:00:58.960532 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 13:00:58.960902 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":469,"threads_started":1}
I20260619 13:00:58.961402 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:58.941224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.1.87.125:43867)
0619 13:00:58.941395 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:58.941402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:58.941796 (+ 394us) server_negotiation.cc:407] Connection header received
0619 13:00:58.942187 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:58.942194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:58.942349 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:58.942467 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:58.943951 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.945096 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.946930 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:58.947586 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:58.947734 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:58.950984 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:58.950990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:58.950995 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:58.951034 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:58.953842 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.954797 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.954803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.954806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.954959 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:58.959485 (+ 4526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:58.959494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:58.959500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:58.959908 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:58.960064 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:58.960649 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 13:00:58.960937 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 13:00:58.963617 7536 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.87.125:43867: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "14031fd252a44b91b756b343b29303c9"
member_type: NON_VOTER
last_known_addr {
host: "127.1.87.125"
port: 43867
}
attrs {
promote: true
}
}
dest_uuid: "9933c1c7f5644e159c996c2efbaabcb6"
cas_config_opid_index: -1
I20260619 13:00:58.964881 7536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } }
I20260619 13:00:58.968853 8094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 14031fd252a44b91b756b343b29303c9 (127.1.87.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } }
I20260619 13:00:58.970661 8040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:58.973278 8094 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.87.125:43867
I20260619 13:00:58.973317 8040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Refusing update from remote peer 9933c1c7f5644e159c996c2efbaabcb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 13:00:58.973805 8094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } }
I20260619 13:00:58.973997 7998 master.cc:537] Master@127.1.87.125:43867 shutting down...
I20260619 13:00:58.974013 8094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:58.974568 8095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:58.976406 8071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:58.976687 8071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } }
I20260619 13:00:58.978026 8071 leader_election.cc:288] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9933c1c7f5644e159c996c2efbaabcb6 (127.1.87.126:33027)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.124.keytab.
Entry for principal oryx/127.1.87.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.124.keytab.
I20260619 13:00:58.978673 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14031fd252a44b91b756b343b29303c9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9933c1c7f5644e159c996c2efbaabcb6" is_pre_election: true
W20260619 13:00:58.979300 7521 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 13:00:58.979794 8014 leader_election.cc:302] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 14031fd252a44b91b756b343b29303c9; no voters: 9933c1c7f5644e159c996c2efbaabcb6
I20260619 13:00:58.980580 8071 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:00:58.981114 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.124:39717
--webserver_interface=127.1.87.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.124.keytab
--principal=oryx/127.1.87.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:33027,127.1.87.125:43867,127.1.87.124:39717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:00:58.991406 7998 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:00:58.991881 7998 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:00:58.992074 7998 tablet_replica.cc:330] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: stopping tablet replica
I20260619 13:00:58.998486 7998 master.cc:552] Master@127.1.87.125:43867 shutdown complete.
I20260619 13:00:58.998585 7998 master_runner.cc:305] Clearing existing system tablet
I20260619 13:00:58.998942 7998 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:58.999047 7998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260619 13:00:58.999814 7521 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 -> Peer 14031fd252a44b91b756b343b29303c9 (127.1.87.125:43867): Couldn't send request to peer 14031fd252a44b91b756b343b29303c9. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260619 13:00:59.001920 7998 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260619 13:00:59.009539 8100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:59.009879 7998 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:59.010217 7998 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.007s
I20260619 13:00:59.010311 7998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal
uuid: "14031fd252a44b91b756b343b29303c9"
format_stamp: "Formatted at 2026-06-19 13:00:58 on dist-test-slave-tpsm"
server_key: "ca807f887f0090cad0f4d9c94a7fb447"
server_key_iv: "58394997a7e6a199c0be44d2204c5b4d"
server_key_version: "encryptionkey@0"
I20260619 13:00:59.014148 7998 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:59.018874 7998 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 13:00:59.019068 7998 log.cc:1196] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 13:00:59.019589 7998 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: Deleting consensus metadata
I20260619 13:00:59.019892 7998 master_runner.cc:315] Copying system tablet from 127.1.87.126:33027
I20260619 13:00:59.022189 7998 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: tablet copy: Beginning tablet copy session from remote peer at address 127.1.87.126:33027
I20260619 13:00:59.039736 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.022964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.1.87.126:33027)
0619 13:00:59.023173 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.023182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.023750 (+ 568us) server_negotiation.cc:407] Connection header received
0619 13:00:59.023876 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.023882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.023969 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.024080 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.024891 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.025606 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.026806 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.027360 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.027488 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.032783 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.032789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.032795 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.032834 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.036476 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.037377 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.037383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.037388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.037529 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.038033 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.038039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.038042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.038420 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.038565 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.039069 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.039374 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:00:59.039798 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.022913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:33674)
0619 13:00:59.023560 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.023586 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.023757 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.024101 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.024106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.024527 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:59.024749 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.024757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.025754 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.025759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.026636 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.026658 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.027502 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.027508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.027711 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.031515 (+ 3804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:59.031538 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:59.032558 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:59.036657 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.036662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.036666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.037247 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.037665 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.037672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.037675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.037917 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.038561 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:59.038568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:59.038785 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.038958 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.039289 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":470,"threads_started":1}
I20260619 13:00:59.041219 7557 tablet_copy_service.cc:140] P 9933c1c7f5644e159c996c2efbaabcb6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 14031fd252a44b91b756b343b29303c9 ({username='oryx', principal='oryx/127.1.87.125@KRBTEST.COM'} at 127.0.0.1:33674)
I20260619 13:00:59.041446 7557 tablet_copy_service.cc:161] P 9933c1c7f5644e159c996c2efbaabcb6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 14031fd252a44b91b756b343b29303c9 at {username='oryx', principal='oryx/127.1.87.125@KRBTEST.COM'} at 127.0.0.1:33674: session id = 14031fd252a44b91b756b343b29303c9-00000000000000000000000000000000
I20260619 13:00:59.045985 7557 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 13:00:59.048516 7998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 13:00:59.055824 7998 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: tablet copy: Starting download of 0 data blocks...
I20260619 13:00:59.056131 7998 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: tablet copy: Starting download of 1 WAL segments...
I20260619 13:00:59.058858 7998 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 13:00:59.063503 7557 tablet_copy_service.cc:342] P 9933c1c7f5644e159c996c2efbaabcb6: Request end of tablet copy session 14031fd252a44b91b756b343b29303c9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.87.125@KRBTEST.COM'} at 127.0.0.1:33674
I20260619 13:00:59.063705 7557 tablet_copy_service.cc:433] P 9933c1c7f5644e159c996c2efbaabcb6: ending tablet copy session 14031fd252a44b91b756b343b29303c9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 14031fd252a44b91b756b343b29303c9
I20260619 13:00:59.074240 7998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:59.076882 8107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:59.087237 7998 server_base.cc:1034] running on GCE node
W20260619 13:00:59.087612 8110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:59.087661 8108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:59.088138 7998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:59.088762 7998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:59.089967 7998 hybrid_clock.cc:648] HybridClock initialized: now 1781874059089911 us; error 35 us; skew 500 ppm
I20260619 13:00:59.090188 7998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:59 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874059, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:59.092128 7998 init.cc:377] Logged in from keytab as oryx/127.1.87.125@KRBTEST.COM (short username oryx)
I20260619 13:00:59.098392 7998 webserver.cc:460] Webserver started at http://127.1.87.125:33277/ using document root <none> and password file <none>
I20260619 13:00:59.098812 7998 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:59.098948 7998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:59.101490 7998 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 13:00:59.103518 8117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:59.103914 7998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 13:00:59.104049 7998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal
uuid: "14031fd252a44b91b756b343b29303c9"
format_stamp: "Formatted at 2026-06-19 13:00:58 on dist-test-slave-tpsm"
server_key: "ca807f887f0090cad0f4d9c94a7fb447"
server_key_iv: "58394997a7e6a199c0be44d2204c5b4d"
server_key_version: "encryptionkey@0"
I20260619 13:00:59.104214 7998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:59.118602 7998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:59.122107 7998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:59.152359 7998 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.125:43867
I20260619 13:00:59.152359 8168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.125:43867 every 8 connection(s)
I20260619 13:00:59.153254 7998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
I20260619 13:00:59.160377 8169 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:00:59.164603 8169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: Bootstrap starting.
I20260619 13:00:59.177883 8169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:00:59.178357 8169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: Bootstrap complete.
I20260619 13:00:59.179184 8169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } }
I20260619 13:00:59.179812 8169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14031fd252a44b91b756b343b29303c9, State: Initialized, Role: LEARNER
I20260619 13:00:59.180305 8169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } }
I20260619 13:00:59.181152 8172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } }
I20260619 13:00:59.181455 8169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:59.181600 8172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:00:59.195827 8183 catalog_manager.cc:1259] Loaded cluster ID: 5cd6dc33edf249409d8ef737f361d0a8
I20260619 13:00:59.195958 8183 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: loading cluster ID for follower catalog manager: success
I20260619 13:00:59.198683 8183 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: acquiring CA information for follower catalog manager: success
I20260619 13:00:59.200938 8183 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 13:00:59.382741 8098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:00:59.383075 8098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:00:59.383227 8098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 13:00:59.385280 7993 heartbeater.cc:503] Master 127.1.87.126:33027 was elected leader, sending a full tablet report...
W20260619 13:00:59.391073 8098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:00:59.391180 8098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:00:59.391266 8098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:00:59.391386 8098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:00:59.391460 8098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:00:59.403427 8098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:33027,127.1.87.125:43867,127.1.87.124:39717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.124.keytab
--principal=oryx/127.1.87.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.124:39717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.87.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:00:59.404775 8098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:00:59.406018 8098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:59.417254 8188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:59.417402 8191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:59.417436 8189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:59.418550 8098 server_base.cc:1034] running on GCE node
I20260619 13:00:59.419257 8098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:59.420593 8098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:59.421789 8098 hybrid_clock.cc:648] HybridClock initialized: now 1781874059421752 us; error 38 us; skew 500 ppm
I20260619 13:00:59.422132 8098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:59 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874059, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:59.425004 8098 init.cc:377] Logged in from keytab as oryx/127.1.87.124@KRBTEST.COM (short username oryx)
I20260619 13:00:59.426385 8098 webserver.cc:460] Webserver started at http://127.1.87.124:41219/ using document root <none> and password file <none>
I20260619 13:00:59.427112 8098 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:59.427251 8098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:59.427592 8098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:00:59.430401 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data/instance:
uuid: "7a303b287ecc4822bd13e022f58cdb82"
format_stamp: "Formatted at 2026-06-19 13:00:59 on dist-test-slave-tpsm"
server_key: "6f5ab1eaa60d39b76f369f4b75336b57"
server_key_iv: "ff051e572eb11830f01563a60db646c1"
server_key_version: "encryptionkey@0"
I20260619 13:00:59.431093 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal/instance:
uuid: "7a303b287ecc4822bd13e022f58cdb82"
format_stamp: "Formatted at 2026-06-19 13:00:59 on dist-test-slave-tpsm"
server_key: "6f5ab1eaa60d39b76f369f4b75336b57"
server_key_iv: "ff051e572eb11830f01563a60db646c1"
server_key_version: "encryptionkey@0"
I20260619 13:00:59.436520 8098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 13:00:59.440199 8198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:59.441557 8098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:00:59.441722 8098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal
uuid: "7a303b287ecc4822bd13e022f58cdb82"
format_stamp: "Formatted at 2026-06-19 13:00:59 on dist-test-slave-tpsm"
server_key: "6f5ab1eaa60d39b76f369f4b75336b57"
server_key_iv: "ff051e572eb11830f01563a60db646c1"
server_key_version: "encryptionkey@0"
I20260619 13:00:59.441958 8098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:59.458086 8098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:59.462014 8098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:00:59.462298 8098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:59.481964 8098 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.124:39717
I20260619 13:00:59.481985 8249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.124:39717 every 8 connection(s)
I20260619 13:00:59.483460 8098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
I20260619 13:00:59.486936 8250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:00:59.493502 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 8098
I20260619 13:00:59.493610 8250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:00:59 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874059, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.124@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
Jun 19 13:00:59 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.124@KRBTEST.COM
I20260619 13:00:59.515790 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.496814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.1.87.126:33027)
0619 13:00:59.496995 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.497002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.497586 (+ 584us) server_negotiation.cc:407] Connection header received
0619 13:00:59.499058 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.499063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.499148 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.499262 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.500642 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.501828 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.504205 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.504906 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.505054 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.509646 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.509654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.509659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.509703 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.512638 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.513530 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.513536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.513541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.513666 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.514078 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.514083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.514087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.514354 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.514502 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.515200 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.515457 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:00:59.516909 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.496728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:33676)
0619 13:00:59.497441 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.497481 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.498715 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.499360 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.499371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.500098 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:59.500472 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.500489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.502009 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.502014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.504028 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.504044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.505160 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.505168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.505324 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.506244 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:59.506266 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:59.509343 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:59.512778 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.512791 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.512804 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.513401 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.513781 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.513786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.513790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.513963 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.514478 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:59.514485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:59.514829 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.515116 (+ 287us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.515658 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":38,"thread_start_us":390,"threads_started":1}
I20260619 13:00:59.516927 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.495737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.1.87.124:39717)
0619 13:00:59.496564 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.496578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.496627 (+ 49us) server_negotiation.cc:407] Connection header received
0619 13:00:59.498698 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.498719 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.499004 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.499232 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.500810 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.502361 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.504205 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.505085 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.505329 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.510587 (+ 5258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.510596 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.510610 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.510639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.513492 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.514185 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.514192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.514194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.514309 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.514690 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.514694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.514696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.515022 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.515371 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.515677 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.515910 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":603,"spinlock_wait_cycles":32384,"thread_start_us":383,"threads_started":1}
I20260619 13:00:59.519121 8250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:00:59 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874059, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.124@KRBTEST.COM for oryx/127.1.87.125@KRBTEST.COM
I20260619 13:00:59.533002 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.520342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.1.87.125:43867)
0619 13:00:59.520877 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.520881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.520906 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:00:59.520958 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.520961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.521019 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.521150 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.521797 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.522742 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.524033 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.524428 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.524527 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.528047 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.528053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.528057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.528087 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.530664 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.531376 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.531380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.531382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.531508 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.531881 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.531885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.531887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.532151 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.532276 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.532578 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.532743 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":306,"threads_started":1}
I20260619 13:00:59.533005 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.520146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.0.0.1:60370)
0619 13:00:59.520470 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.520494 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.520622 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.521199 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.521203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.521531 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:59.521673 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.521680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.522864 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.522867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.523894 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.523905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.524566 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.524570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.524726 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.525310 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:59.525327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:59.527851 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:59.530798 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.530801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.530804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.531202 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.531618 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.531622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.531624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.531758 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.532256 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:59.532259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:59.532385 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.532516 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.532729 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 13:00:59.534081 8250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:00:59 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874059, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.124@KRBTEST.COM for oryx/127.1.87.124@KRBTEST.COM
I20260619 13:00:59.546455 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.534780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:39717 (local address 127.0.0.1:53996)
0619 13:00:59.534915 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.534929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.535003 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.535359 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.535368 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.535681 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:59.535817 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.535824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.536853 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.536857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.537712 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.537722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.538356 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.538360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.538514 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.539004 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:59.539019 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:59.541122 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:59.544066 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.544069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.544072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.544397 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.544748 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.544752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.544755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.544918 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.545494 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:59.545497 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:59.545611 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.545753 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.545944 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 13:00:59.546495 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.534930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.1.87.124:39717)
0619 13:00:59.535098 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.535102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.535117 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:00:59.535155 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.535158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.535209 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.535291 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.535944 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.536739 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.537826 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.538219 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.538320 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.541319 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.541323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.541327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.541354 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.543929 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.544504 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.544508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.544510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.544617 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.545041 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.545045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.545047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.545385 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.545499 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.545845 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.546072 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":73}
I20260619 13:00:59.552003 8250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Bootstrap starting.
I20260619 13:00:59.554579 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.549069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.1.87.125:43867)
0619 13:00:59.549245 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.549265 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.549561 (+ 296us) server_negotiation.cc:407] Connection header received
0619 13:00:59.549767 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.549772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.549868 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.549978 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:00:59.550391 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.551086 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.552800 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.553603 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.553774 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.553988 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.554204 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.554330 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 13:00:59.554662 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.548993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.0.0.1:60384)
0619 13:00:59.549466 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.549481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.549632 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.550018 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.550024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.550050 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:00:59.550258 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.550266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.551220 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.551225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.552660 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.552671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.553809 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.553815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.554014 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.554064 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.554169 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.554314 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":271,"threads_started":1}
I20260619 13:00:59.556146 8250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Neither blocks nor log segments found. Creating new log.
I20260619 13:00:59.556171 8144 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 13:00:59.556764 8172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9933c1c7f5644e159c996c2efbaabcb6. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } }
I20260619 13:00:59.557026 8172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:00:59.557535 8250 log.cc:824] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Log is configured to *not* fsync() on all Append() calls
I20260619 13:00:59.560276 8250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: No bootstrap required, opened a new log
I20260619 13:00:59.568536 8250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } }
I20260619 13:00:59.568850 8250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:00:59.568928 8250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a303b287ecc4822bd13e022f58cdb82, State: Initialized, Role: FOLLOWER
I20260619 13:00:59.569419 8250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } }
I20260619 13:00:59.570218 8259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } } }
I20260619 13:00:59.570766 8259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:00:59.571288 8250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:00:59.580101 8270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:00:59.580250 8270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:00:59.584586 8098 master_runner.cc:428] Detected that this master 7a303b287ecc4822bd13e022f58cdb82 is joining an existing cluster
I20260619 13:00:59.584741 8098 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.87.124:39717
I20260619 13:00:59.598444 8259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:00:59.599433 8259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } }
I20260619 13:00:59.602800 8259 leader_election.cc:288] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9933c1c7f5644e159c996c2efbaabcb6 (127.1.87.126:33027), 14031fd252a44b91b756b343b29303c9 (127.1.87.125:43867)
I20260619 13:00:59.603775 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a303b287ecc4822bd13e022f58cdb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9933c1c7f5644e159c996c2efbaabcb6" is_pre_election: true
I20260619 13:00:59.604012 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a303b287ecc4822bd13e022f58cdb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14031fd252a44b91b756b343b29303c9" is_pre_election: true
I20260619 13:00:59.604753 8199 leader_election.cc:302] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a303b287ecc4822bd13e022f58cdb82; no voters: 14031fd252a44b91b756b343b29303c9, 9933c1c7f5644e159c996c2efbaabcb6
I20260619 13:00:59.605439 8259 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 13:00:59.613037 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.589753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.1.87.126:33027)
0619 13:00:59.589981 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.589987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.590784 (+ 797us) server_negotiation.cc:407] Connection header received
0619 13:00:59.590866 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.590874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.590973 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.591084 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.593110 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.594239 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.596968 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.597678 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.597802 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.605763 (+ 7961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.605770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.605775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.605815 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.609324 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.610134 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.610141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.610145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.610295 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.610800 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.610808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.610813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.611178 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.611345 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.612373 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.612596 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:00:59.613171 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.590864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.1.87.125:43867)
0619 13:00:59.591049 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.591054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.593662 (+ 2608us) server_negotiation.cc:407] Connection header received
0619 13:00:59.593910 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.593915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.593995 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.594063 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.596322 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.597906 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.601301 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.602239 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.602417 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.605948 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.605954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.605958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.606005 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.609857 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.610669 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.610674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.610677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.610819 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.611360 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.611368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.611374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.611731 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.612442 (+ 711us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.612585 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.612833 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:00:59.613241 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.590630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.0.0.1:60388)
0619 13:00:59.593542 (+ 2912us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.593572 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.593764 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.594313 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.594320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.595269 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:59.596063 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.596081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.600050 (+ 3969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.600055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.601137 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.601148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.602771 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.602781 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.603022 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.604444 (+ 1422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:59.604466 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:59.605752 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:59.610010 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.610015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.610019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.610490 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.610945 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.610950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.610954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.611171 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.611866 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:59.611874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:59.612081 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.612228 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.612760 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2619,"mutex_wait_us":227,"thread_start_us":2478,"threads_started":1}
I20260619 13:00:59.613963 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.589775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:33692)
0619 13:00:59.590474 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.590503 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.590628 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.591296 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.591304 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.592309 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:59.592805 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.592822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.594427 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.594433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.596789 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.596804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.598038 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.598045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.598230 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.599287 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:59.599337 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:59.605177 (+ 5840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:59.609493 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.609498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.609503 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.609994 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.610426 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.610431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.610434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.610667 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.611407 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:59.611415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:59.611750 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.611905 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.613535 (+ 1630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":372,"threads_started":1}
I20260619 13:00:59.621258 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.594025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:39717 (local address 127.0.0.1:53998)
0619 13:00:59.603365 (+ 9340us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.603390 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.603533 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.603949 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.603953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.604653 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:59.605170 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.605179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.606600 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.606608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.608790 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.608805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.610107 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.610115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.610310 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.611069 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:59.611106 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:59.612386 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:59.618440 (+ 6054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.618448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.618453 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.618823 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.619230 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.619235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.619239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.619419 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.620135 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:59.620142 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:59.620311 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.620462 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.620767 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9171,"thread_start_us":9004,"threads_started":1}
I20260619 13:00:59.621929 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.593778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.1.87.124:39717)
0619 13:00:59.593924 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.593929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.603451 (+ 9522us) server_negotiation.cc:407] Connection header received
0619 13:00:59.603648 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.603652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.603715 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.603775 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.605311 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.606389 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.608935 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.609710 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.609802 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.614293 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.614299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.614304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.614337 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.618329 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.618938 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.618943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.618946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.619110 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.619534 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.619539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.619543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.619935 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.620064 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.620574 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.621654 (+ 1080us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":814,"server-negotiator.queue_time_us":61}
I20260619 13:00:59.624325 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 7598
Jun 19 13:00:59 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.126@KRBTEST.COM for oryx/127.1.87.124@KRBTEST.COM
I20260619 13:00:59.635799 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.619296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:39717 (local address 127.0.0.1:54008)
0619 13:00:59.619495 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.619513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.619611 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.620757 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.620764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.621182 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:59.621343 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.621355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.622971 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.622978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.624122 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.624135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.625118 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.625127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.625308 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.626061 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:59.626084 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:59.628910 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:59.632969 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.632975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.632981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.633524 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.633954 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.633963 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.633968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.634128 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.634762 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:59.634768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:59.634986 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.635143 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.635451 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 13:00:59.635874 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.619645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.1.87.124:39717)
0619 13:00:59.620379 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.620382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.620398 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:00:59.620501 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.620505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.620580 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.620736 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.621495 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.622834 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.624263 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.624944 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.625619 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.629151 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.629158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.629163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.629197 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.632827 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.633670 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.633676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.633679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.633837 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.634263 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.634272 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.634276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.634617 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.634757 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.635249 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.635467 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":468,"threads_started":1}
Jun 19 13:00:59 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.65@KRBTEST.COM
I20260619 13:00:59.637631 7536 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.87.124:39717: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82"
member_type: NON_VOTER
last_known_addr {
host: "127.1.87.124"
port: 39717
}
attrs {
promote: true
}
}
dest_uuid: "9933c1c7f5644e159c996c2efbaabcb6"
cas_config_opid_index: 5
I20260619 13:00:59.639966 7536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } }
I20260619 13:00:59.641609 8256 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7a303b287ecc4822bd13e022f58cdb82 (127.1.87.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.642256 8144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 LEARNER]: Refusing update from remote peer 9933c1c7f5644e159c996c2efbaabcb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 13:00:59.643566 8225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:00:59.645228 8256 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.87.124:39717
I20260619 13:00:59.646287 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.625848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.1.87.65:42863)
0619 13:00:59.626523 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.626532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.626560 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:00:59.626629 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.626637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.626879 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.627040 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.628630 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.629615 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.631027 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.631635 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.631761 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.637321 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.637339 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.637359 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.637400 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.641253 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.642284 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.642293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.642297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.642448 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.642907 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.642912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.642916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.643384 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.645266 (+ 1882us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.645572 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.645871 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":604,"threads_started":1}
I20260619 13:00:59.647710 7721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:00:59.651644 8225 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 FOLLOWER]: Refusing update from remote peer 9933c1c7f5644e159c996c2efbaabcb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 13:00:59.656414 8256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 13:00:59.656606 8284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.657094 8284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:59.657063 8256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:00:59.663605 8098 master.cc:537] Master@127.1.87.124:39717 shutting down...
I20260619 13:00:59.674885 8144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7a303b287ecc4822bd13e022f58cdb82 (127.1.87.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.679520 8172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.679792 8172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:00:59.680748 8282 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: attempting to promote NON_VOTER 14031fd252a44b91b756b343b29303c9 to VOTER
I20260619 13:00:59.680713 8225 raft_consensus.cc:858] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } }
W20260619 13:00:59.680904 8225 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } }
I20260619 13:00:59.681577 8282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } }
I20260619 13:00:59.690084 8144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 LEARNER]: Refusing update from remote peer 9933c1c7f5644e159c996c2efbaabcb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 13:00:59.696612 8281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 13:00:59.698771 7521 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 -> Peer 7a303b287ecc4822bd13e022f58cdb82 (127.1.87.124:39717): Couldn't send request to peer 7a303b287ecc4822bd13e022f58cdb82. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 13:00:59.699676 8284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 14031fd252a44b91b756b343b29303c9 (127.1.87.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.703943 8282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.704244 8282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:59.704490 8282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.704768 8282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:00:59.705451 8144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 14031fd252a44b91b756b343b29303c9 (127.1.87.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.709685 8172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.709913 8172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:00:59.720103 8225 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 14031fd252a44b91b756b343b29303c9 (127.1.87.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } }
I20260619 13:00:59.725621 8290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } }
I20260619 13:00:59.725893 8290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 13:00:59.726177 8259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9933c1c7f5644e159c996c2efbaabcb6. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } }
I20260619 13:00:59.726372 8259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 13:00:59.727257 8293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: NON_VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: true } } }
I20260619 13:00:59.727453 8098 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 13:00:59.727558 8293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 13:00:59.727918 8098 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 13:00:59.728128 8098 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: stopping tablet replica
I20260619 13:00:59.746259 8098 master.cc:552] Master@127.1.87.124:39717 shutdown complete.
I20260619 13:00:59.746362 8098 master_runner.cc:305] Clearing existing system tablet
I20260619 13:00:59.746737 8098 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:59.746826 8098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:59.749861 8098 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260619 13:00:59.751567 8295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:59.751966 8098 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:59.752226 8098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 13:00:59.752420 8098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal
uuid: "7a303b287ecc4822bd13e022f58cdb82"
format_stamp: "Formatted at 2026-06-19 13:00:59 on dist-test-slave-tpsm"
server_key: "6f5ab1eaa60d39b76f369f4b75336b57"
server_key_iv: "ff051e572eb11830f01563a60db646c1"
server_key_version: "encryptionkey@0"
I20260619 13:00:59.757040 8098 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:00:59.763778 8098 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 13:00:59.763942 8098 log.cc:1196] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 13:00:59.764438 8098 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Deleting consensus metadata
I20260619 13:00:59.764672 8098 master_runner.cc:315] Copying system tablet from 127.1.87.126:33027
I20260619 13:00:59.767117 8098 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: tablet copy: Beginning tablet copy session from remote peer at address 127.1.87.126:33027
I20260619 13:00:59.780839 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 7598
I20260619 13:00:59.787299 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.767882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:33702)
0619 13:00:59.772173 (+ 4291us) negotiation.cc:107] Waiting for socket to connect
0619 13:00:59.772195 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:00:59.772362 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:00:59.772724 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:00:59.772729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:00:59.773240 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:00:59.773575 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.773588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.775146 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.775151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.776681 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:00:59.776695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.777667 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.777676 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:00:59.777923 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.778756 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:00:59.778788 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:00:59.780017 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:00:59.784300 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.784308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.784315 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.784866 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.785287 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:00:59.785294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:00:59.785305 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:00:59.785495 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:00:59.786217 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:00:59.786222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:00:59.786390 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 13:00:59.786550 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 13:00:59.786824 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3765,"thread_start_us":3634,"threads_started":1}
I20260619 13:00:59.787449 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:00:59.767954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.1.87.126:33027)
0619 13:00:59.768195 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 13:00:59.768202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:00:59.772308 (+ 4106us) server_negotiation.cc:407] Connection header received
0619 13:00:59.772497 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:00:59.772505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:00:59.772592 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:00:59.772702 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:00:59.773722 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.774962 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.776838 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:00:59.777474 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:00:59.777595 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:00:59.780202 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:00:59.780208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:00:59.780214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:00:59.780252 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:00:59.784163 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.785015 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.785021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.785024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.785180 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:00:59.785656 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:00:59.785661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:00:59.785664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:00:59.785980 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:00:59.786157 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 13:00:59.786704 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 13:00:59.786975 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260619 13:00:59.788282 7557 tablet_copy_service.cc:140] P 9933c1c7f5644e159c996c2efbaabcb6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7a303b287ecc4822bd13e022f58cdb82 ({username='oryx', principal='oryx/127.1.87.124@KRBTEST.COM'} at 127.0.0.1:33702)
I20260619 13:00:59.788458 7557 tablet_copy_service.cc:161] P 9933c1c7f5644e159c996c2efbaabcb6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7a303b287ecc4822bd13e022f58cdb82 at {username='oryx', principal='oryx/127.1.87.124@KRBTEST.COM'} at 127.0.0.1:33702: session id = 7a303b287ecc4822bd13e022f58cdb82-00000000000000000000000000000000
I20260619 13:00:59.792761 7557 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 13:00:59.794607 8098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 13:00:59.801329 8098 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: tablet copy: Starting download of 0 data blocks...
I20260619 13:00:59.801608 8098 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: tablet copy: Starting download of 1 WAL segments...
I20260619 13:00:59.801988 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:42863
--local_ip_for_outbound_sockets=127.1.87.65
--tserver_master_addrs=127.1.87.126:33027
--webserver_port=45365
--webserver_interface=127.1.87.65
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab
--principal=oryx/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.87.126:33027,127.1.87.125:43867
--tserver_master_addrs=127.1.87.126:33027,127.1.87.125:43867,127.1.87.124:39717 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:00:59.803833 8098 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 13:00:59.815613 7557 tablet_copy_service.cc:342] P 9933c1c7f5644e159c996c2efbaabcb6: Request end of tablet copy session 7a303b287ecc4822bd13e022f58cdb82-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.87.124@KRBTEST.COM'} at 127.0.0.1:33702
I20260619 13:00:59.815770 7557 tablet_copy_service.cc:433] P 9933c1c7f5644e159c996c2efbaabcb6: ending tablet copy session 7a303b287ecc4822bd13e022f58cdb82-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7a303b287ecc4822bd13e022f58cdb82
I20260619 13:00:59.832908 8098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:00:59.835866 8303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:59.839954 8304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:00:59.844048 8306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:00:59.844251 8098 server_base.cc:1034] running on GCE node
I20260619 13:00:59.844750 8098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:00:59.845346 8098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:00:59.846546 8098 hybrid_clock.cc:648] HybridClock initialized: now 1781874059846504 us; error 34 us; skew 500 ppm
I20260619 13:00:59.846773 8098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:00:59 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874059, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:00:59.848922 8098 init.cc:377] Logged in from keytab as oryx/127.1.87.124@KRBTEST.COM (short username oryx)
I20260619 13:00:59.856393 8098 webserver.cc:460] Webserver started at http://127.1.87.124:36797/ using document root <none> and password file <none>
I20260619 13:00:59.856721 8098 fs_manager.cc:359] Metadata directory not provided
I20260619 13:00:59.856825 8098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:00:59.860028 8098 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 13:00:59.862004 8313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:00:59.862516 8098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 13:00:59.862679 8098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal
uuid: "7a303b287ecc4822bd13e022f58cdb82"
format_stamp: "Formatted at 2026-06-19 13:00:59 on dist-test-slave-tpsm"
server_key: "6f5ab1eaa60d39b76f369f4b75336b57"
server_key_iv: "ff051e572eb11830f01563a60db646c1"
server_key_version: "encryptionkey@0"
I20260619 13:00:59.862913 8098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:00:59.914325 8098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:00:59.918323 8098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:00:59.939839 8098 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.124:39717
I20260619 13:00:59.940160 8364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.124:39717 every 8 connection(s)
I20260619 13:00:59.941016 8098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
I20260619 13:00:59.946422 8365 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:00:59.950507 8365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Bootstrap starting.
I20260619 13:00:59.968685 8365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:00:59.969225 8365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Bootstrap complete.
I20260619 13:00:59.970252 8365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } }
I20260619 13:00:59.970531 8365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a303b287ecc4822bd13e022f58cdb82, State: Initialized, Role: LEARNER
I20260619 13:00:59.970767 8365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } }
I20260619 13:00:59.971601 8368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:00:59.972047 8368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:00:59.972208 8365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:00:59.984449 8379 catalog_manager.cc:1259] Loaded cluster ID: 5cd6dc33edf249409d8ef737f361d0a8
I20260619 13:00:59.984556 8379 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: loading cluster ID for follower catalog manager: success
I20260619 13:00:59.988008 8379 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: acquiring CA information for follower catalog manager: success
I20260619 13:00:59.990442 8379 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:01:00.073791 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.065265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:39717 (local address 127.0.0.1:54024)
0619 13:01:00.065557 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:00.065577 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:00.065767 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:00.066706 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:00.066712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:00.066754 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:00.066936 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.066946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.069016 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.069022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.070873 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.070889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.072807 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.072814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.073035 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.073077 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 13:01:00.073198 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 13:01:00.073410 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260619 13:01:00.073807 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.065394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.1.87.124:39717)
0619 13:01:00.066332 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:00.066338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:00.066362 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:01:00.066432 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:00.066437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:00.066521 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:00.066642 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:00.067098 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.068837 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.071043 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.072620 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.072783 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.073063 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:00.073235 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 13:01:00.073407 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":599,"threads_started":1}
I20260619 13:01:00.074961 8340 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 13:01:00.075511 8368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9933c1c7f5644e159c996c2efbaabcb6. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:01:00.075836 8368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: This master's current role is: LEARNER
W20260619 13:01:00.166282 8302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:00.166574 8302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:00.166745 8302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:00.174841 8302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:00.174984 8302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:00.175146 8302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:01:00.184927 8302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab
--principal=oryx/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:42863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=45365
--tserver_master_addrs=127.1.87.126:33027,127.1.87.125:43867,127.1.87.124:39717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:00.186240 8302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:00.187650 8302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:00.201269 8385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:00.201236 8388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:00.201663 8386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:00.202677 8302 server_base.cc:1034] running on GCE node
I20260619 13:01:00.203281 8302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:00.204387 8302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:00.205643 8302 hybrid_clock.cc:648] HybridClock initialized: now 1781874060205549 us; error 88 us; skew 500 ppm
I20260619 13:01:00.205976 8302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874060, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:00.209494 8302 init.cc:377] Logged in from keytab as oryx/127.1.87.65@KRBTEST.COM (short username oryx)
I20260619 13:01:00.210924 8302 webserver.cc:460] Webserver started at http://127.1.87.65:45365/ using document root <none> and password file <none>
I20260619 13:01:00.211656 8302 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:00.211769 8302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:00.216938 8302 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 13:01:00.219461 8395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:00.220759 8302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:01:00.220887 8302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "a1b9fc73b1714ddabd5d5cd927adacf1"
format_stamp: "Formatted at 2026-06-19 13:00:57 on dist-test-slave-tpsm"
server_key: "b7212c6f6f59a3e24da8543242b47967"
server_key_iv: "7f68ea78b5b3414f7da9990f352b43cd"
server_key_version: "encryptionkey@0"
I20260619 13:01:00.221781 8302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:00.236371 8302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:00.240499 8302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:00.240736 8302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:00.242156 8302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:00.243538 8302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:00.243620 8302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:00.243692 8302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:00.243747 8302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:00.277673 8302 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:42863
I20260619 13:01:00.277760 8507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:42863 every 8 connection(s)
I20260619 13:01:00.279055 8302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:01:00.284967 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 8302
I20260619 13:01:00.285308 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 7732
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874060, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874060, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.125@KRBTEST.COM
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874060, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.124@KRBTEST.COM
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.66@KRBTEST.COM
I20260619 13:01:00.308306 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.284542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:48869 (local address 127.1.87.126:33027)
0619 13:01:00.284703 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:00.284708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:00.285350 (+ 642us) server_negotiation.cc:407] Connection header received
0619 13:01:00.288228 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:00.288236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:00.288318 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:00.288385 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:00.289840 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.290942 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.293545 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.294175 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.294269 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.300615 (+ 6346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:00.300624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:00.300632 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:00.300693 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:00.304169 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.305311 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.305318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.305321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.305471 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.306083 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.306089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.306093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.306554 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:00.306715 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:00.307751 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0619 13:01:00.307974 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:01:00.308669 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.286298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.1.87.66:43129)
0619 13:01:00.287327 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:00.287337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:00.287367 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:01:00.287429 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:00.287436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:00.287668 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:00.287819 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:00.289910 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.291397 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.293652 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.294472 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.294576 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.300809 (+ 6233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:00.300824 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:00.300841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:00.300889 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:00.304659 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.305549 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.305558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.305561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.305702 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.306115 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.306119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.306123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.306613 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:00.306882 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:00.308016 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0619 13:01:00.308263 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":723,"threads_started":1}
I20260619 13:01:00.309154 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.281840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:58903 (local address 127.1.87.125:43867)
0619 13:01:00.282299 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:00.282303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:00.283244 (+ 941us) server_negotiation.cc:407] Connection header received
0619 13:01:00.286861 (+ 3617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:00.286866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:00.286971 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:00.287079 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:00.289840 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.290942 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.293780 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.294421 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.294514 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.299835 (+ 5321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:00.299842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:00.299865 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:00.299903 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:00.305241 (+ 5338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.306039 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.306045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.306051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.306193 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.306709 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.306715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.306719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.307053 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:00.307167 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:00.308114 (+ 947us) server_negotiation.cc:299] Negotiation successful
0619 13:01:00.308332 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":266,"threads_started":1}
I20260619 13:01:00.309758 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.283768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:40961 (local address 127.1.87.124:39717)
0619 13:01:00.283945 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:00.283951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:00.284717 (+ 766us) server_negotiation.cc:407] Connection header received
0619 13:01:00.286969 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:00.286978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:00.287065 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:00.287150 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:00.289855 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.290965 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.293786 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.294387 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.294478 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.300169 (+ 5691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:00.300175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:00.300180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:00.300219 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:00.304159 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.305317 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.305322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.305325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.305590 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.306090 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.306097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.306100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.306881 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:00.307007 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:00.308241 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0619 13:01:00.308491 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:01:00.310082 7855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:00.310084 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.284123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.65:48869)
0619 13:01:00.285250 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:00.285263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:00.288060 (+ 2797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:00.288574 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:00.288580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:00.289122 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:00.289658 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.289674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.291139 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.291144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.293380 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.293394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.294563 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.294571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.294900 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.295797 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:00.295849 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:00.300395 (+ 4546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:00.304337 (+ 3942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.304348 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.304367 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.305167 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.305615 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.305620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.305626 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.305954 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.306807 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:00.306815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:00.307237 (+ 422us) client_negotiation.cc:769] Sending connection context
0619 13:01:00.308541 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0619 13:01:00.308940 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":178,"spinlock_wait_cycles":24576,"thread_start_us":910,"threads_started":1}
I20260619 13:01:00.310719 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.283553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:39717 (local address 127.1.87.65:40961)
0619 13:01:00.284613 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:00.284631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:00.286570 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:00.287529 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:00.287541 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:00.288515 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:00.289678 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.289693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.291142 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.291148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.293639 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.293651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.294788 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.294794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.295107 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.295938 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:00.295970 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:00.299935 (+ 3965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:00.304398 (+ 4463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.304405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.304409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.305179 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.305736 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.305741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.305744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.305961 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.307084 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:00.307092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:00.307250 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 13:01:00.307602 (+ 352us) client_negotiation.cc:240] Negotiation successful
0619 13:01:00.310283 (+ 2681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":283,"spinlock_wait_cycles":13056,"thread_start_us":254,"threads_started":1}
I20260619 13:01:00.311676 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.282333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.1.87.65:58903)
0619 13:01:00.283102 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:00.283141 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:00.286547 (+ 3406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:00.287530 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:00.287541 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:00.288538 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:00.289650 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.289674 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.291139 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.291144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.293646 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.293659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.294801 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.294808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.295025 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.295797 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:00.295849 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:00.299499 (+ 3650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:00.305402 (+ 5903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.305409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.305413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.305909 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.306408 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.306414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.306417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.306557 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.307236 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:00.307240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:00.307698 (+ 458us) client_negotiation.cc:769] Sending connection context
0619 13:01:00.310618 (+ 2920us) client_negotiation.cc:240] Negotiation successful
0619 13:01:00.311145 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":307,"spinlock_wait_cycles":34944,"thread_start_us":317,"threads_started":1}
I20260619 13:01:00.312477 8509 heartbeater.cc:347] Connected to a master server at 127.1.87.124:39717
I20260619 13:01:00.312618 8508 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33027
I20260619 13:01:00.312805 8509 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:00.312824 8508 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:00.313714 8510 heartbeater.cc:347] Connected to a master server at 127.1.87.125:43867
I20260619 13:01:00.313830 8510 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:00.314292 8510 heartbeater.cc:511] Master 127.1.87.125:43867 requested a full tablet report, sending...
I20260619 13:01:00.314325 8508 heartbeater.cc:511] Master 127.1.87.126:33027 requested a full tablet report, sending...
I20260619 13:01:00.315109 8509 heartbeater.cc:511] Master 127.1.87.124:39717 requested a full tablet report, sending...
I20260619 13:01:00.316402 8134 ts_manager.cc:194] Registered new tserver with Master: a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863)
I20260619 13:01:00.317118 8330 ts_manager.cc:194] Registered new tserver with Master: a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863)
I20260619 13:01:00.317533 7536 ts_manager.cc:194] Re-registered known tserver with Master: a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863)
I20260619 13:01:00.318594 7536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:48869
I20260619 13:01:00.402446 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 7732
I20260619 13:01:00.418545 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:43129
--local_ip_for_outbound_sockets=127.1.87.66
--tserver_master_addrs=127.1.87.126:33027
--webserver_port=41613
--webserver_interface=127.1.87.66
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab
--principal=oryx/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.87.126:33027,127.1.87.125:43867
--tserver_master_addrs=127.1.87.126:33027,127.1.87.125:43867,127.1.87.124:39717 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:01:00.688550 8282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: NON_VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: true } } }
I20260619 13:01:00.688876 8282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:00.690493 8520 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: attempting to promote NON_VOTER 7a303b287ecc4822bd13e022f58cdb82 to VOTER
I20260619 13:01:00.691035 8520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } }
I20260619 13:01:00.692800 8340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 LEARNER]: Refusing update from remote peer 9933c1c7f5644e159c996c2efbaabcb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 13:01:00.692962 8144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Refusing update from remote peer 9933c1c7f5644e159c996c2efbaabcb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 13:01:00.693331 8521 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 13:01:00.693789 8282 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 13:01:00.696225 8521 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a303b287ecc4822bd13e022f58cdb82 (127.1.87.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:00.697136 8340 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a303b287ecc4822bd13e022f58cdb82 (127.1.87.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:00.700518 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:00.700191 8144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a303b287ecc4822bd13e022f58cdb82 (127.1.87.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:00.700780 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:00.701035 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:00.701265 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:00.701469 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:00.701653 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:00.705729 8525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:00.706128 8525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:01:00.707739 8524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9933c1c7f5644e159c996c2efbaabcb6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:00.708058 8524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 13:01:00.744568 8519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:00.744850 8519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:00.744943 8519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:00.752627 8519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:00.752743 8519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:00.752878 8519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:01:00.763270 8519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab
--principal=oryx/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:43129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=41613
--tserver_master_addrs=127.1.87.126:33027,127.1.87.125:43867,127.1.87.124:39717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:00.764544 8519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:00.765843 8519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:00.779289 8533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:00.779562 8530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:00.779561 8531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:00.780544 8519 server_base.cc:1034] running on GCE node
I20260619 13:01:00.781163 8519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:00.782121 8519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:00.783355 8519 hybrid_clock.cc:648] HybridClock initialized: now 1781874060783280 us; error 30 us; skew 500 ppm
I20260619 13:01:00.783674 8519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874060, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:00.786569 8519 init.cc:377] Logged in from keytab as oryx/127.1.87.66@KRBTEST.COM (short username oryx)
I20260619 13:01:00.788043 8519 webserver.cc:460] Webserver started at http://127.1.87.66:41613/ using document root <none> and password file <none>
I20260619 13:01:00.788802 8519 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:00.788916 8519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:00.794083 8519 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 13:01:00.796705 8540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:00.798209 8519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:01:00.798377 8519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "259e01b38a444737934f88d0c1ccad01"
format_stamp: "Formatted at 2026-06-19 13:00:57 on dist-test-slave-tpsm"
server_key: "c248764194da5348bfe84219f0a341b1"
server_key_iv: "402f447fe0381166507d6029e960f9a3"
server_key_version: "encryptionkey@0"
I20260619 13:01:00.799506 8519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:00.829033 8519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:00.833433 8519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:00.833747 8519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:00.834710 8519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:00.836199 8519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:00.836284 8519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:00.836434 8519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:00.836535 8519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:00.870909 8519 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:43129
I20260619 13:01:00.871004 8652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:43129 every 8 connection(s)
I20260619 13:01:00.872466 8519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:01:00.878228 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 8519
I20260619 13:01:00.878533 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 7865
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874060, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.124@KRBTEST.COM
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874060, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:01:00.906266 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.875379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:44073 (local address 127.1.87.124:39717)
0619 13:01:00.875874 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:00.875879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:00.877169 (+ 1290us) server_negotiation.cc:407] Connection header received
0619 13:01:00.880461 (+ 3292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:00.880470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:00.880568 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:00.880672 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:00.883378 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.884765 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.887678 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.888520 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.888637 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.898938 (+ 10301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:00.898944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:00.898949 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:00.898990 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:00.902388 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.903399 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.903404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.903408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.903546 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.904164 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.904170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.904173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.904524 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:00.904668 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:00.905651 (+ 983us) server_negotiation.cc:299] Negotiation successful
0619 13:01:00.905875 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
I20260619 13:01:00.908017 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.875859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:39717 (local address 127.1.87.66:44073)
0619 13:01:00.876996 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:00.877057 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:00.880123 (+ 3066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:00.881305 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:00.881320 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:00.882246 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:00.883133 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.883157 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.885036 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.885043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.887467 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.887486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.889051 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.889060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.889462 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.890926 (+ 1464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:00.890967 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:00.898691 (+ 7724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:00.902529 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.902537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.902563 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.903260 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.903680 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.903686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.903690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.903961 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.904813 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:00.904819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:00.905289 (+ 470us) client_negotiation.cc:769] Sending connection context
0619 13:01:00.905570 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 13:01:00.906096 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":3,"thread_start_us":272,"threads_started":1}
I20260619 13:01:00.909629 8654 heartbeater.cc:347] Connected to a master server at 127.1.87.124:39717
I20260619 13:01:00.909920 8654 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:00.910930 8654 heartbeater.cc:511] Master 127.1.87.124:39717 requested a full tablet report, sending...
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874060, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.125@KRBTEST.COM
I20260619 13:01:00.912883 8330 ts_manager.cc:194] Registered new tserver with Master: 259e01b38a444737934f88d0c1ccad01 (127.1.87.66:43129)
I20260619 13:01:00.914016 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.877309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.66:44173)
0619 13:01:00.877971 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:00.877990 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:00.884899 (+ 6909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:00.890617 (+ 5718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:00.890626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:00.892970 (+ 2344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:00.893283 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.893295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.895124 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.895133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.897239 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.897256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.899951 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.899959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.900126 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.900659 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:00.900687 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:00.904355 (+ 3668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:00.908702 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.908707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.908712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.909906 (+ 1194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.911205 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.911210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.911213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.911385 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.913055 (+ 1670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:00.913060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:00.913213 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 13:01:00.913353 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 13:01:00.913628 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":332,"threads_started":1}
I20260619 13:01:00.914175 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.877481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:44173 (local address 127.1.87.126:33027)
0619 13:01:00.890236 (+ 12755us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:00.890246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:00.890270 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:01:00.890343 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:00.890350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:00.890448 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:00.890578 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:00.893472 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.894919 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.898908 (+ 3989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.899558 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.899697 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.904536 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:00.904545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:00.904551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:00.904590 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:00.908196 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.910895 (+ 2699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.910900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.910903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.911057 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.911520 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.911526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.911529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.911877 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:00.912004 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:00.913496 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0619 13:01:00.913718 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12605,"thread_start_us":12517,"threads_started":1}
I20260619 13:01:00.915261 8653 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33027
I20260619 13:01:00.915445 8653 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:00.915822 8653 heartbeater.cc:511] Master 127.1.87.126:33027 requested a full tablet report, sending...
I20260619 13:01:00.916764 7536 ts_manager.cc:194] Re-registered known tserver with Master: 259e01b38a444737934f88d0c1ccad01 (127.1.87.66:43129)
I20260619 13:01:00.917557 7536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:44173
Jun 19 13:01:00 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.67@KRBTEST.COM
I20260619 13:01:00.922096 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.880037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:33965 (local address 127.1.87.125:43867)
0619 13:01:00.898302 (+ 18265us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:00.898309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:00.901516 (+ 3207us) server_negotiation.cc:407] Connection header received
0619 13:01:00.901590 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:00.901595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:00.901685 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:00.903037 (+ 1352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:00.903353 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.904598 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.906862 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.907863 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.908002 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.912612 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:00.912618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:00.912622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:00.912689 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:00.917998 (+ 5309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.918875 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.918881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.918885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.920263 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.920714 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.920717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.920720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.920999 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:00.921163 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:00.921514 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 13:01:00.921717 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18149,"thread_start_us":18274,"threads_started":1}
I20260619 13:01:00.922125 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.880065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.1.87.66:33965)
0619 13:01:00.899773 (+ 19708us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:00.899794 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:00.900035 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:00.901876 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:00.901883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:00.902444 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:00.902883 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.902896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.904758 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.904763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.906497 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:00.906519 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.908452 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.908461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:00.908630 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.909236 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:00.909276 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:00.912408 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:00.918170 (+ 5762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.918176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.918180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.918728 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.920410 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:00.920418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:00.920423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:00.920595 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:00.921128 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:00.921133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:00.921278 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 13:01:00.921416 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 13:01:00.921724 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19578,"thread_start_us":19447,"threads_started":1}
I20260619 13:01:00.922900 8655 heartbeater.cc:347] Connected to a master server at 127.1.87.125:43867
I20260619 13:01:00.923038 8655 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:00.923501 8655 heartbeater.cc:511] Master 127.1.87.125:43867 requested a full tablet report, sending...
I20260619 13:01:00.924645 8134 ts_manager.cc:194] Registered new tserver with Master: 259e01b38a444737934f88d0c1ccad01 (127.1.87.66:43129)
I20260619 13:01:00.926265 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:00.879503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.1.87.67:36091)
0619 13:01:00.900657 (+ 21154us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:00.900667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:00.900708 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:01:00.900778 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:00.900788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:00.901069 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:00.901262 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:00.903493 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.909315 (+ 5822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.910842 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:00.912354 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:00.912501 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:00.919117 (+ 6616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:00.919132 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:00.919150 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:00.919207 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:00.922545 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.923381 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.923389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.923392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.923547 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:00.924006 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:00.924010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:00.924012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:00.924404 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:00.924704 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:00.925670 (+ 966us) server_negotiation.cc:299] Negotiation successful
0619 13:01:00.925935 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20956,"thread_start_us":21042,"threads_started":1}
I20260619 13:01:00.927621 7988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:01.010139 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 7865
I20260619 13:01:01.026232 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:36091
--local_ip_for_outbound_sockets=127.1.87.67
--tserver_master_addrs=127.1.87.126:33027
--webserver_port=35793
--webserver_interface=127.1.87.67
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab
--principal=oryx/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.87.126:33027,127.1.87.125:43867
--tserver_master_addrs=127.1.87.126:33027,127.1.87.125:43867,127.1.87.124:39717 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:01:01.321698 8508 heartbeater.cc:503] Master 127.1.87.126:33027 was elected leader, sending a full tablet report...
W20260619 13:01:01.347081 8666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:01.347419 8666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:01.347579 8666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:01.354985 8666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:01.355132 8666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:01.355304 8666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:01:01.364912 8666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab
--principal=oryx/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:36091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=35793
--tserver_master_addrs=127.1.87.126:33027,127.1.87.125:43867,127.1.87.124:39717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:01.366174 8666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:01.367460 8666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:01.380594 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:01.380628 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:01.380594 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:01.381669 8666 server_base.cc:1034] running on GCE node
I20260619 13:01:01.382205 8666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:01.383102 8666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:01.384328 8666 hybrid_clock.cc:648] HybridClock initialized: now 1781874061384275 us; error 64 us; skew 500 ppm
I20260619 13:01:01.384620 8666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:01 dist-test-slave-tpsm krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874061, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:01.387619 8666 init.cc:377] Logged in from keytab as oryx/127.1.87.67@KRBTEST.COM (short username oryx)
I20260619 13:01:01.389125 8666 webserver.cc:460] Webserver started at http://127.1.87.67:35793/ using document root <none> and password file <none>
I20260619 13:01:01.389755 8666 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:01.389860 8666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:01.394845 8666 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:01:01.397397 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:01.398671 8666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:01:01.398826 8666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "f1707fe2ccdf4c9fbb06aa52a092446b"
format_stamp: "Formatted at 2026-06-19 13:00:58 on dist-test-slave-tpsm"
server_key: "3f5ea1532211d1c6c1181300b5b0e796"
server_key_iv: "04a76b0c62ab1799a3aa555c4fadfc70"
server_key_version: "encryptionkey@0"
I20260619 13:01:01.399930 8666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:01.422712 8666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:01.427369 8666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:01.427626 8666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:01.428490 8666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:01.429755 8666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:01.429840 8666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:01.429915 8666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:01.429968 8666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:01.461100 8666 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:36091
I20260619 13:01:01.461156 8793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:36091 every 8 connection(s)
I20260619 13:01:01.462476 8666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:01:01.465117 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 8666
Jun 19 13:01:01 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874061, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.124@KRBTEST.COM
Jun 19 13:01:01 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874061, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.125@KRBTEST.COM
I20260619 13:01:01.500432 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.466525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:58131 (local address 127.1.87.124:39717)
0619 13:01:01.466931 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.466938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.466962 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:01:01.469615 (+ 2653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.469624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.469738 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.469830 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.472276 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.473845 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.476543 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.477283 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.477508 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.487270 (+ 9762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.487279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.487287 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.487362 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.491362 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.497459 (+ 6097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.497465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.497469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.497628 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.498174 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.498179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.498182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.498535 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.498687 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.499841 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.500075 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":497,"threads_started":1}
I20260619 13:01:01.501577 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.465749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:39717 (local address 127.1.87.67:58131)
0619 13:01:01.466497 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.466550 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.469240 (+ 2690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.470369 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.470383 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.471203 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:01.472070 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.472098 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.474014 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.474020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.476342 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.476362 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.477938 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.477947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.478352 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.480386 (+ 2034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:01.480424 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:01.487006 (+ 6582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:01.492826 (+ 5820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:01.492839 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:01.492863 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:01.497324 (+ 4461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:01.497802 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:01.497806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:01.497809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:01.498044 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:01.498680 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:01.498689 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:01.499208 (+ 519us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.499565 (+ 357us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.500133 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":310,"threads_started":1}
I20260619 13:01:01.503242 8795 heartbeater.cc:347] Connected to a master server at 127.1.87.124:39717
I20260619 13:01:01.503610 8795 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:01.503885 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.469079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.1.87.126:33027)
0619 13:01:01.475742 (+ 6663us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.475750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.475774 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:01:01.475847 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.475855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.475948 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.476909 (+ 961us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.481797 (+ 4888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.482914 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.486774 (+ 3860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.487451 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.487574 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.492023 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.492041 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.492046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.492086 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.496644 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.498011 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.498018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.498023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.498215 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.501970 (+ 3755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.501975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.501978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.502301 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.502522 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.502901 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.503532 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6512,"thread_start_us":6407,"threads_started":1}
I20260619 13:01:01.504560 8795 heartbeater.cc:511] Master 127.1.87.124:39717 requested a full tablet report, sending...
I20260619 13:01:01.506295 8330 ts_manager.cc:194] Registered new tserver with Master: f1707fe2ccdf4c9fbb06aa52a092446b (127.1.87.67:36091)
I20260619 13:01:01.509337 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.467086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.1.87.67:60275)
0619 13:01:01.478562 (+ 11476us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.478580 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.478701 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.479256 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.479261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.479905 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:01.480213 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.480227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.483081 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.483087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.484703 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.484720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.485846 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.485853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.486019 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.487713 (+ 1694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:01.487741 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:01.497013 (+ 9272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:01.501940 (+ 4927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:01.501948 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:01.501953 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:01.504467 (+ 2514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:01.505006 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:01.505010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:01.505013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:01.505146 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:01.508296 (+ 3150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:01.508300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:01.508464 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.508601 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.508898 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11333,"thread_start_us":11157,"threads_started":1}
I20260619 13:01:01.509537 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.470870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.1.87.124:39717)
0619 13:01:01.472642 (+ 1772us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.472650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.472670 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:01:01.472728 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.472736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.472824 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.472935 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.483939 (+ 11004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.485828 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.491672 (+ 5844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.492457 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.492601 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.498886 (+ 6285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.498893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.498898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.498940 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.505947 (+ 7007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.506811 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.506816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.506819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.506991 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.507455 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.507462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.507465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.507808 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.508027 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.508926 (+ 899us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.509135 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1613,"thread_start_us":1485,"threads_started":1}
I20260619 13:01:01.510577 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.467359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:60275 (local address 127.1.87.125:43867)
0619 13:01:01.477443 (+ 10084us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.477450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.478801 (+ 1351us) server_negotiation.cc:407] Connection header received
0619 13:01:01.478852 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.478858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.478928 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.479025 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.480397 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.481625 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.484864 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.485508 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.485632 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.497242 (+ 11610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.497248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.497253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.497288 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.501144 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.504683 (+ 3539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.504689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.504692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.504867 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.506484 (+ 1617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.506492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.506498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.508081 (+ 1583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.508232 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.509981 (+ 1749us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.510204 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9955,"thread_start_us":9812,"threads_started":1}
I20260619 13:01:01.511508 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.470073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.1.87.125:43867)
0619 13:01:01.477746 (+ 7673us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.477750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.477763 (+ 13us) server_negotiation.cc:407] Connection header received
0619 13:01:01.477806 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.477809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.477866 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.477985 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.480720 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.481950 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.486853 (+ 4903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.492036 (+ 5183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.492724 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.495923 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.495930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.495935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.495968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.507279 (+ 11311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.508121 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.508128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.508134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.508301 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.508739 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.508744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.508747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.509458 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.510905 (+ 1447us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.511028 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.511189 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7591,"thread_start_us":7491,"threads_started":1}
I20260619 13:01:01.512179 8796 heartbeater.cc:347] Connected to a master server at 127.1.87.125:43867
I20260619 13:01:01.512327 8796 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:01.513049 8796 heartbeater.cc:511] Master 127.1.87.125:43867 requested a full tablet report, sending...
I20260619 13:01:01.514010 8133 ts_manager.cc:194] Registered new tserver with Master: f1707fe2ccdf4c9fbb06aa52a092446b (127.1.87.67:36091)
Jun 19 13:01:01 dist-test-slave-tpsm krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874061, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:01:01.508549 7536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:01:01.518857 7536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:01:01.522408 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.478890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:40749 (local address 127.1.87.126:33027)
0619 13:01:01.495610 (+ 16720us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.495617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.496292 (+ 675us) server_negotiation.cc:407] Connection header received
0619 13:01:01.496351 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.496357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.496425 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.496521 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.499865 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.501038 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.505360 (+ 4322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.507516 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.507629 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.516545 (+ 8916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.516551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.516554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.516585 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.519851 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.520477 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.520482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.520485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.520643 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.521005 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.521010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.521013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.521402 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.521556 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.521929 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.522076 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16044,"thread_start_us":16552,"threads_started":1}
I20260619 13:01:01.522482 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.478639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.67:40749)
0619 13:01:01.496029 (+ 17390us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.496051 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.496183 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.498726 (+ 2543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.498733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.499352 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:01.499692 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.499712 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.501362 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.501368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.505033 (+ 3665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.505048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.512194 (+ 7146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.512200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.512402 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.513060 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:01.513082 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:01.516320 (+ 3238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:01.519995 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:01.519999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:01.520001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:01.520341 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:01.520754 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:01.520757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:01.520760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:01.520883 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:01.521538 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:01.521543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:01.521687 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.521843 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.522109 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17251,"thread_start_us":22505,"threads_started":1}
I20260619 13:01:01.523402 8794 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33027
I20260619 13:01:01.523543 8794 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:01.523980 8794 heartbeater.cc:511] Master 127.1.87.126:33027 requested a full tablet report, sending...
I20260619 13:01:01.525213 7537 ts_manager.cc:194] Re-registered known tserver with Master: f1707fe2ccdf4c9fbb06aa52a092446b (127.1.87.67:36091)
I20260619 13:01:01.526257 7537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:40749
I20260619 13:01:01.551282 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.538514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42863 (local address 127.0.0.1:50272)
0619 13:01:01.540072 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.540097 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.540234 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.541514 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.541519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.541579 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:01.541954 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.541967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.544256 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.544265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.547327 (+ 3062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.547349 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.549889 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.549899 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.550257 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.550317 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.550419 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.550616 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1407,"mutex_wait_us":78,"thread_start_us":6131,"threads_started":1}
I20260619 13:01:01.553073 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.538743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.1.87.65:42863)
0619 13:01:01.540778 (+ 2035us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.540790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.540823 (+ 33us) server_negotiation.cc:407] Connection header received
0619 13:01:01.540934 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.540945 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.541261 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.541506 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:01.542185 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.544084 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.547565 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.549608 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.551984 (+ 2376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.552284 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.552369 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.552607 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1822,"thread_start_us":1686,"threads_started":1}
I20260619 13:01:01.554936 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.539688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.1.87.66:43129)
0619 13:01:01.543775 (+ 4087us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.543784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.545279 (+ 1495us) server_negotiation.cc:407] Connection header received
0619 13:01:01.545570 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.545579 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.545877 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.546073 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:01.547116 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.548740 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.552195 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.553694 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.553799 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.553989 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.554535 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.554703 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3928,"thread_start_us":3799,"threads_started":1}
I20260619 13:01:01.556114 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.539056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:43129 (local address 127.0.0.1:55890)
0619 13:01:01.545160 (+ 6104us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.545182 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.545377 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.546388 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.546395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.546429 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:01.546895 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.546909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.548906 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.548911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.551971 (+ 3060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.551991 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.554190 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.554196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.554327 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.554376 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.555135 (+ 759us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.555258 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5732,"mutex_wait_us":30,"thread_start_us":724,"threads_started":1}
I20260619 13:01:01.559667 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.545007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36091 (local address 127.0.0.1:39856)
0619 13:01:01.551602 (+ 6595us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.551625 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.551740 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.554420 (+ 2680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.554427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.554459 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:01.554621 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.554632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.556206 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.556213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.557667 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.557679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.558921 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.558926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.559103 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.559139 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.559223 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.559377 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6421,"thread_start_us":8870,"threads_started":1}
I20260619 13:01:01.559765 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.545365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.1.87.67:36091)
0619 13:01:01.553837 (+ 8472us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.553846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.553880 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:01:01.553977 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.553984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.554223 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.554415 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:01.554754 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.556073 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.557822 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.558758 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.558912 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.559096 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.559281 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.559447 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8282,"thread_start_us":8606,"threads_started":1}
I20260619 13:01:01.559333 8588 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1aa4b221ad432e812db73b86fce79f (DEFAULT_TABLE table=test-table [id=ce00702c05cd440ea41ba1cdc57338fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:01.558710 8443 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1aa4b221ad432e812db73b86fce79f (DEFAULT_TABLE table=test-table [id=ce00702c05cd440ea41ba1cdc57338fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:01.560673 8588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1aa4b221ad432e812db73b86fce79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:01.561133 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1aa4b221ad432e812db73b86fce79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:01.562434 8729 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1aa4b221ad432e812db73b86fce79f (DEFAULT_TABLE table=test-table [id=ce00702c05cd440ea41ba1cdc57338fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:01.563495 8729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1aa4b221ad432e812db73b86fce79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:01.578735 8829 tablet_bootstrap.cc:492] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: Bootstrap starting.
I20260619 13:01:01.580889 8828 tablet_bootstrap.cc:492] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: Bootstrap starting.
I20260619 13:01:01.582901 8829 tablet_bootstrap.cc:654] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:01.584209 8829 log.cc:824] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:01.584723 8830 tablet_bootstrap.cc:492] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: Bootstrap starting.
I20260619 13:01:01.586241 8828 tablet_bootstrap.cc:654] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:01.587628 8828 log.cc:824] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:01.590092 8830 tablet_bootstrap.cc:654] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:01.590904 8829 tablet_bootstrap.cc:492] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: No bootstrap required, opened a new log
I20260619 13:01:01.590905 8828 tablet_bootstrap.cc:492] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: No bootstrap required, opened a new log
I20260619 13:01:01.591015 8830 log.cc:824] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:01.591172 8829 ts_tablet_manager.cc:1397] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260619 13:01:01.591172 8828 ts_tablet_manager.cc:1397] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260619 13:01:01.593341 8830 tablet_bootstrap.cc:492] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: No bootstrap required, opened a new log
I20260619 13:01:01.593601 8830 ts_tablet_manager.cc:1397] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 13:01:01.600203 8830 raft_consensus.cc:348] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
I20260619 13:01:01.600620 8830 raft_consensus.cc:374] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:01.600715 8830 raft_consensus.cc:729] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259e01b38a444737934f88d0c1ccad01, State: Initialized, Role: FOLLOWER
I20260619 13:01:01.601406 8830 consensus_queue.cc:260] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
I20260619 13:01:01.601949 8828 raft_consensus.cc:348] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
I20260619 13:01:01.602430 8828 raft_consensus.cc:374] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:01.602520 8828 raft_consensus.cc:729] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1707fe2ccdf4c9fbb06aa52a092446b, State: Initialized, Role: FOLLOWER
I20260619 13:01:01.602348 8829 raft_consensus.cc:348] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
I20260619 13:01:01.602804 8829 raft_consensus.cc:374] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:01.602903 8829 raft_consensus.cc:729] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b9fc73b1714ddabd5d5cd927adacf1, State: Initialized, Role: FOLLOWER
I20260619 13:01:01.603511 8653 heartbeater.cc:503] Master 127.1.87.126:33027 was elected leader, sending a full tablet report...
I20260619 13:01:01.604681 8830 ts_tablet_manager.cc:1428] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 13:01:01.603214 8828 consensus_queue.cc:260] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
I20260619 13:01:01.607069 8829 consensus_queue.cc:260] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
I20260619 13:01:01.609326 8794 heartbeater.cc:503] Master 127.1.87.126:33027 was elected leader, sending a full tablet report...
I20260619 13:01:01.609812 8829 ts_tablet_manager.cc:1428] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
I20260619 13:01:01.612267 8828 ts_tablet_manager.cc:1428] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: Time spent starting tablet: real 0.021s user 0.010s sys 0.005s
I20260619 13:01:01.617820 8834 raft_consensus.cc:482] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:01.618116 8834 raft_consensus.cc:504] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
I20260619 13:01:01.619621 8834 leader_election.cc:288] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863), f1707fe2ccdf4c9fbb06aa52a092446b (127.1.87.67:36091)
I20260619 13:01:01.624028 8835 raft_consensus.cc:482] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:01.624284 8835 raft_consensus.cc:504] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
W20260619 13:01:01.625658 8656 tablet.cc:2369] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:01:01.626114 8835 leader_election.cc:288] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863), 259e01b38a444737934f88d0c1ccad01 (127.1.87.66:43129)
I20260619 13:01:01.627792 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.619840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42863 (local address 127.1.87.66:59063)
0619 13:01:01.620470 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.620488 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.620673 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.621172 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.621178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.621207 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:01.621615 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.621630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.622864 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.622870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.624980 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.624993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.626809 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.626817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.627002 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.627041 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.627154 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.627359 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":3,"thread_start_us":321,"threads_started":1}
I20260619 13:01:01.627846 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.619914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:59063 (local address 127.1.87.65:42863)
0619 13:01:01.620084 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.620089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.620559 (+ 470us) server_negotiation.cc:407] Connection header received
0619 13:01:01.620848 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.620855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.620927 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.620984 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:01.621798 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.622722 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.625166 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.626436 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.626529 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.626628 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.627190 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.627624 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:01:01.628208 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.620746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:46335 (local address 127.1.87.67:36091)
0619 13:01:01.621251 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.621257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.621602 (+ 345us) server_negotiation.cc:407] Connection header received
0619 13:01:01.621749 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.621757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.621827 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.621890 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:01.622408 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.623375 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.625292 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.626609 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.626739 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.626846 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.627530 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.627653 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 13:01:01.629091 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.620551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36091 (local address 127.1.87.66:46335)
0619 13:01:01.621498 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.621512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.621634 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.621927 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.621931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.621953 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:01.622245 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.622256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.623498 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.623502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.625149 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.625161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.627094 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.627100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.627340 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.627370 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.628567 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.628703 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":1,"thread_start_us":719,"threads_started":1}
I20260619 13:01:01.629184 8463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1aa4b221ad432e812db73b86fce79f" candidate_uuid: "259e01b38a444737934f88d0c1ccad01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" is_pre_election: true
I20260619 13:01:01.630165 8463 raft_consensus.cc:2454] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259e01b38a444737934f88d0c1ccad01 in term 0.
I20260619 13:01:01.630892 8541 leader_election.cc:302] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259e01b38a444737934f88d0c1ccad01, a1b9fc73b1714ddabd5d5cd927adacf1; no voters:
I20260619 13:01:01.631237 8834 raft_consensus.cc:2781] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:01:01.631400 8834 raft_consensus.cc:482] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:01:01.631503 8834 raft_consensus.cc:3037] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:01.632021 8749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1aa4b221ad432e812db73b86fce79f" candidate_uuid: "259e01b38a444737934f88d0c1ccad01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" is_pre_election: true
I20260619 13:01:01.632449 8749 raft_consensus.cc:2454] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259e01b38a444737934f88d0c1ccad01 in term 0.
I20260619 13:01:01.634089 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.626529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42863 (local address 127.1.87.67:58901)
0619 13:01:01.626736 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.626752 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.626833 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.628393 (+ 1560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.628398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.628433 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:01.628637 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.628648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.630141 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.630147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.631961 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.631975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.633398 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.633404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.633598 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.633620 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.633727 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.633851 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 13:01:01.634119 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.626928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:58901 (local address 127.1.87.65:42863)
0619 13:01:01.628065 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.628070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.628088 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:01:01.628136 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.628140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.628252 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.628768 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:01.628918 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.630012 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.632153 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.633247 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.633381 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.633474 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.633754 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.633866 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":2157,"threads_started":1}
I20260619 13:01:01.634213 8834 raft_consensus.cc:504] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
I20260619 13:01:01.634881 8834 leader_election.cc:288] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [CANDIDATE]: Term 1 election: Requested vote from peers a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863), f1707fe2ccdf4c9fbb06aa52a092446b (127.1.87.67:36091)
I20260619 13:01:01.635340 8749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1aa4b221ad432e812db73b86fce79f" candidate_uuid: "259e01b38a444737934f88d0c1ccad01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b"
I20260619 13:01:01.635294 8463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1aa4b221ad432e812db73b86fce79f" candidate_uuid: "259e01b38a444737934f88d0c1ccad01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1"
I20260619 13:01:01.635517 8749 raft_consensus.cc:3037] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:01.635545 8463 raft_consensus.cc:3037] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:01.635769 8462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1aa4b221ad432e812db73b86fce79f" candidate_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" is_pre_election: true
I20260619 13:01:01.637972 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.627630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:43129 (local address 127.1.87.67:58995)
0619 13:01:01.627896 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:01.627914 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:01.628019 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:01.628798 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:01.628802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:01.628826 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:01.628982 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.628991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.630828 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.630833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.632765 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:01.632777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.636904 (+ 4127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.636913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:01.637203 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.637248 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 13:01:01.637344 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 13:01:01.637488 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260619 13:01:01.638439 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.628236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:58995 (local address 127.1.87.66:43129)
0619 13:01:01.628430 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.628436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.628451 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:01:01.628504 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.628508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.628569 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.628660 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:01.629238 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.630690 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.635432 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.636610 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.636730 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.636811 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.638155 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.638278 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:01:01.639142 8749 raft_consensus.cc:2454] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259e01b38a444737934f88d0c1ccad01 in term 1.
I20260619 13:01:01.639143 8463 raft_consensus.cc:2454] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259e01b38a444737934f88d0c1ccad01 in term 1.
I20260619 13:01:01.639094 8608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1aa4b221ad432e812db73b86fce79f" candidate_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259e01b38a444737934f88d0c1ccad01" is_pre_election: true
I20260619 13:01:01.639575 8608 raft_consensus.cc:2379] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1707fe2ccdf4c9fbb06aa52a092446b in current term 1: Already voted for candidate 259e01b38a444737934f88d0c1ccad01 in this term.
I20260619 13:01:01.639648 8541 leader_election.cc:302] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259e01b38a444737934f88d0c1ccad01, f1707fe2ccdf4c9fbb06aa52a092446b; no voters:
I20260619 13:01:01.639926 8834 raft_consensus.cc:2781] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:01.640516 8685 leader_election.cc:302] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1707fe2ccdf4c9fbb06aa52a092446b; no voters: 259e01b38a444737934f88d0c1ccad01, a1b9fc73b1714ddabd5d5cd927adacf1
I20260619 13:01:01.640554 8834 raft_consensus.cc:686] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 1 LEADER]: Becoming Leader. State: Replica: 259e01b38a444737934f88d0c1ccad01, State: Running, Role: LEADER
I20260619 13:01:01.641132 8835 raft_consensus.cc:2726] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 13:01:01.641096 8834 consensus_queue.cc:237] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } }
I20260619 13:01:01.646557 7537 catalog_manager.cc:5495] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259e01b38a444737934f88d0c1ccad01 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "259e01b38a444737934f88d0c1ccad01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } health_report { overall_health: UNKNOWN } } }
I20260619 13:01:01.675091 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.669472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.1.87.66:43129)
0619 13:01:01.669667 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.669674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.669766 (+ 92us) server_negotiation.cc:407] Connection header received
0619 13:01:01.670003 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.670007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.670090 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.670249 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:01:01.670615 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.671403 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.672926 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.673362 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.673512 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.673973 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:01:01.674492 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:01:01.674644 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.674708 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.674784 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:01:01.684110 8463 raft_consensus.cc:1261] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [term 1 FOLLOWER]: Refusing update from remote peer 259e01b38a444737934f88d0c1ccad01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:01.684190 8749 raft_consensus.cc:1261] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 FOLLOWER]: Refusing update from remote peer 259e01b38a444737934f88d0c1ccad01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:01.684957 8840 consensus_queue.cc:1035] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:01.685415 8834 consensus_queue.cc:1035] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:01.715446 7537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:01.715757 7537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:01.723048 7537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9933c1c7f5644e159c996c2efbaabcb6: Sending DeleteTablet for 3 replicas of tablet 2f1aa4b221ad432e812db73b86fce79f
I20260619 13:01:01.724233 8588 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f1aa4b221ad432e812db73b86fce79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:01 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:55890
I20260619 13:01:01.725297 8443 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f1aa4b221ad432e812db73b86fce79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:01 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:50272
I20260619 13:01:01.726071 8729 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f1aa4b221ad432e812db73b86fce79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:01 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:39856
I20260619 13:01:01.726343 8853 tablet_replica.cc:330] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: stopping tablet replica
I20260619 13:01:01.726761 8853 raft_consensus.cc:2229] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:01:01.727399 8853 raft_consensus.cc:2258] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:01.729476 8853 ts_tablet_manager.cc:1905] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:01.736672 8853 ts_tablet_manager.cc:1918] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:01.736838 8853 log.cc:1196] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/2f1aa4b221ad432e812db73b86fce79f
I20260619 13:01:01.737418 8853 ts_tablet_manager.cc:1939] T 2f1aa4b221ad432e812db73b86fce79f P 259e01b38a444737934f88d0c1ccad01: Deleting consensus metadata
I20260619 13:01:01.739567 7524 catalog_manager.cc:4841] TS 259e01b38a444737934f88d0c1ccad01 (127.1.87.66:43129): tablet 2f1aa4b221ad432e812db73b86fce79f (table test-table [id=ce00702c05cd440ea41ba1cdc57338fa]) successfully deleted
I20260619 13:01:01.743439 8855 tablet_replica.cc:330] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: stopping tablet replica
I20260619 13:01:01.743883 8855 raft_consensus.cc:2229] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:01.744432 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 7505
I20260619 13:01:01.745776 7567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:01.746680 8855 raft_consensus.cc:2258] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:01.749358 8855 ts_tablet_manager.cc:1905] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:01.754405 8854 tablet_replica.cc:330] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: stopping tablet replica
I20260619 13:01:01.754848 8854 raft_consensus.cc:2229] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:01.755371 8854 raft_consensus.cc:2258] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:01.757607 8854 ts_tablet_manager.cc:1905] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:01.761190 8855 ts_tablet_manager.cc:1918] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:01.761412 8855 log.cc:1196] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/2f1aa4b221ad432e812db73b86fce79f
I20260619 13:01:01.762002 8855 ts_tablet_manager.cc:1939] T 2f1aa4b221ad432e812db73b86fce79f P f1707fe2ccdf4c9fbb06aa52a092446b: Deleting consensus metadata
I20260619 13:01:01.765007 8854 ts_tablet_manager.cc:1918] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:01.765221 8854 log.cc:1196] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/2f1aa4b221ad432e812db73b86fce79f
I20260619 13:01:01.765741 8854 ts_tablet_manager.cc:1939] T 2f1aa4b221ad432e812db73b86fce79f P a1b9fc73b1714ddabd5d5cd927adacf1: Deleting consensus metadata
I20260619 13:01:01.769816 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.729243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.1.87.125:43867)
0619 13:01:01.729473 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.729482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.742388 (+ 12906us) server_negotiation.cc:407] Connection header received
0619 13:01:01.743294 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.743303 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.743446 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.743551 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.746484 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.748050 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.753949 (+ 5899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.754516 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.754645 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.762596 (+ 7951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.762603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.762607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.762647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.766661 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.767515 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.767521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.767526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.767701 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.768181 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.768185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.768188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.768546 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.768795 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.769181 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.769415 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:01:01.786127 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.742534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.1.87.124:39717)
0619 13:01:01.742781 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.742790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.769896 (+ 27106us) server_negotiation.cc:407] Connection header received
0619 13:01:01.770132 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.770138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.770232 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.770302 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.772186 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.773309 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.775185 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.775881 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.775977 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.779445 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.779452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.779457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.779497 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.783082 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.783898 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.783904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.783908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.784071 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.784506 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.784512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.784516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.784887 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.785147 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.785489 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.785728 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:01:01.876670 7521 catalog_manager.cc:4841] TS a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863): tablet 2f1aa4b221ad432e812db73b86fce79f (table test-table [id=ce00702c05cd440ea41ba1cdc57338fa]) successfully deleted
I20260619 13:01:01.877933 7521 catalog_manager.cc:4841] TS f1707fe2ccdf4c9fbb06aa52a092446b (127.1.87.67:36091): tablet 2f1aa4b221ad432e812db73b86fce79f (table test-table [id=ce00702c05cd440ea41ba1cdc57338fa]) successfully deleted
I20260619 13:01:01.878544 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 7505
I20260619 13:01:01.926457 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.910853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.1.87.125:43867)
0619 13:01:01.911060 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.911069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.911095 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:01:01.911403 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.911410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.911498 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.911576 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.913584 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.914344 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.916260 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.916659 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.916807 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.919921 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.919927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.919934 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.919973 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.923323 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.924109 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.924116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.924119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.924270 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.924702 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.924707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.924710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.925078 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.925324 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.925701 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.926060 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:01:01.928630 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:01.912026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.1.87.124:39717)
0619 13:01:01.912199 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:01.912205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:01.912223 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:01:01.912281 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:01.912285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:01.912376 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:01.912486 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:01.914768 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.916027 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.917888 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:01.918569 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:01.918722 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:01.921965 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:01.921973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:01.921979 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:01.922019 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:01.925665 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.926603 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.926610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.926614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.926759 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:01.927158 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:01.927163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:01.927167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:01.927516 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:01.927771 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:01.928118 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 13:01:01.928334 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:01:02.707299 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:02.706340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.67:57507)
0619 13:01:02.706840 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:02.707014 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
W20260619 13:01:02.708246 8794 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:33027 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
I20260619 13:01:02.713845 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:02.713048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.66:60575)
0619 13:01:02.713493 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:02.713630 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":230,"threads_started":1}
W20260619 13:01:02.714257 8653 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:33027 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
I20260619 13:01:02.726195 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:02.725514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.65:55735)
0619 13:01:02.725901 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:02.726002 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":186,"threads_started":1}
W20260619 13:01:02.726559 8508 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:33027 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
I20260619 13:01:03.222921 8867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9933c1c7f5644e159c996c2efbaabcb6)
I20260619 13:01:03.223111 8867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } }
I20260619 13:01:03.223692 8868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9933c1c7f5644e159c996c2efbaabcb6)
I20260619 13:01:03.223984 8867 leader_election.cc:288] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9933c1c7f5644e159c996c2efbaabcb6 (127.1.87.126:33027), 14031fd252a44b91b756b343b29303c9 (127.1.87.125:43867)
I20260619 13:01:03.223922 8868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } }
I20260619 13:01:03.224975 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.224133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54324)
0619 13:01:03.224530 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.224764 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":201,"threads_started":1}
I20260619 13:01:03.225644 8868 leader_election.cc:288] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9933c1c7f5644e159c996c2efbaabcb6 (127.1.87.126:33027), 7a303b287ecc4822bd13e022f58cdb82 (127.1.87.124:39717)
W20260619 13:01:03.225617 8314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
I20260619 13:01:03.227254 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.226853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54338)
0619 13:01:03.227002 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.227099 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:01:03.227439 8314 leader_election.cc:334] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9933c1c7f5644e159c996c2efbaabcb6 (127.1.87.126:33027): Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
I20260619 13:01:03.228132 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.224946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54326)
0619 13:01:03.227756 (+ 2810us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.227926 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2717,"thread_start_us":2565,"threads_started":1}
W20260619 13:01:03.228678 8118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
I20260619 13:01:03.232915 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.231856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54354)
0619 13:01:03.232605 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.232740 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":507,"threads_started":1}
W20260619 13:01:03.233074 8118 leader_election.cc:334] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9933c1c7f5644e159c996c2efbaabcb6 (127.1.87.126:33027): Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
I20260619 13:01:03.235669 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.224714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:43867 (local address 127.0.0.1:50468)
0619 13:01:03.225265 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.225287 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:03.225453 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:03.228590 (+ 3137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:03.228595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:03.228619 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:03.228856 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.228870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.231113 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.231118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.233486 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.233500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.235101 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.235106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.235224 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.235247 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:01:03.235378 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 13:01:03.235482 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":520,"threads_started":1}
I20260619 13:01:03.235859 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.227736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.1.87.125:43867)
0619 13:01:03.228283 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:03.228289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:03.228307 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:01:03.228369 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:03.228374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:03.228440 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:03.228992 (+ 552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:03.229340 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.230933 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.233636 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.234944 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.235062 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.235186 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:03.235397 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 13:01:03.235511 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":422,"thread_start_us":258,"threads_started":1}
I20260619 13:01:03.236506 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a303b287ecc4822bd13e022f58cdb82" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "14031fd252a44b91b756b343b29303c9" is_pre_election: true
I20260619 13:01:03.236730 8144 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a303b287ecc4822bd13e022f58cdb82 in term 1.
I20260619 13:01:03.237177 8314 leader_election.cc:302] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14031fd252a44b91b756b343b29303c9, 7a303b287ecc4822bd13e022f58cdb82; no voters: 9933c1c7f5644e159c996c2efbaabcb6
I20260619 13:01:03.237465 8867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 13:01:03.237612 8867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9933c1c7f5644e159c996c2efbaabcb6)
I20260619 13:01:03.237720 8867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:01:03.237914 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.228442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:39717 (local address 127.0.0.1:53330)
0619 13:01:03.228609 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.228643 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:03.228786 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:03.232031 (+ 3245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:03.232037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:03.232073 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:03.232375 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.232390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.233809 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.233819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.235798 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.235810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.237230 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.237236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.237392 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.237422 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:01:03.237521 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 13:01:03.237698 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 13:01:03.238103 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.228456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.1.87.124:39717)
0619 13:01:03.231782 (+ 3326us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:03.231786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:03.231803 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:01:03.231843 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:03.231847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:03.231895 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:03.232010 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:03.232567 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.233693 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.235987 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.237053 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.237183 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.237258 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:03.237576 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 13:01:03.237728 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3222,"thread_start_us":3059,"threads_started":1}
I20260619 13:01:03.238564 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14031fd252a44b91b756b343b29303c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7a303b287ecc4822bd13e022f58cdb82" is_pre_election: true
I20260619 13:01:03.240200 8867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } }
I20260619 13:01:03.240626 8340 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 14031fd252a44b91b756b343b29303c9 in current term 2: Already voted for candidate 7a303b287ecc4822bd13e022f58cdb82 in this term.
I20260619 13:01:03.240780 8867 leader_election.cc:288] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [CANDIDATE]: Term 2 election: Requested vote from peers 9933c1c7f5644e159c996c2efbaabcb6 (127.1.87.126:33027), 14031fd252a44b91b756b343b29303c9 (127.1.87.125:43867)
I20260619 13:01:03.241308 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.240916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54356)
0619 13:01:03.241055 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.241139 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 13:01:03.241554 8118 leader_election.cc:302] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14031fd252a44b91b756b343b29303c9; no voters: 7a303b287ecc4822bd13e022f58cdb82, 9933c1c7f5644e159c996c2efbaabcb6
I20260619 13:01:03.241525 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a303b287ecc4822bd13e022f58cdb82" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "14031fd252a44b91b756b343b29303c9"
I20260619 13:01:03.241778 8144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:01:03.242146 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.241849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54364)
0619 13:01:03.241983 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.242047 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 13:01:03.242278 8314 leader_election.cc:334] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9933c1c7f5644e159c996c2efbaabcb6 (127.1.87.126:33027): Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
I20260619 13:01:03.243934 8144 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a303b287ecc4822bd13e022f58cdb82 in term 2.
I20260619 13:01:03.244125 8868 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:01:03.244359 8314 leader_election.cc:302] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14031fd252a44b91b756b343b29303c9, 7a303b287ecc4822bd13e022f58cdb82; no voters: 9933c1c7f5644e159c996c2efbaabcb6
I20260619 13:01:03.244616 8867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:01:03.244994 8867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [term 2 LEADER]: Becoming Leader. State: Replica: 7a303b287ecc4822bd13e022f58cdb82, State: Running, Role: LEADER
I20260619 13:01:03.245345 8867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } }
I20260619 13:01:03.246810 8877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a303b287ecc4822bd13e022f58cdb82. Latest consensus state: current_term: 2 leader_uuid: "7a303b287ecc4822bd13e022f58cdb82" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:03.247103 8877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:03.247792 8879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:01:03.251085 8879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:01:03.251879 8879 catalog_manager.cc:1259] Loaded cluster ID: 5cd6dc33edf249409d8ef737f361d0a8
I20260619 13:01:03.251947 8879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:01:03.252951 8879 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:01:03.253672 8879 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Loaded TSK: 0
I20260619 13:01:03.254386 8879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:01:03.267382 8330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:01:03.274266 8330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:01:03.278636 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.278233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54384)
0619 13:01:03.278405 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.278507 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 13:01:03.279146 8144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [term 2 FOLLOWER]: Refusing update from remote peer 7a303b287ecc4822bd13e022f58cdb82: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 13:01:03.279716 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.279263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54388)
0619 13:01:03.279452 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.279542 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 13:01:03.279892 8314 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 -> Peer 9933c1c7f5644e159c996c2efbaabcb6 (127.1.87.126:33027): Couldn't send request to peer 9933c1c7f5644e159c996c2efbaabcb6. Status: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 13:01:03.279948 8877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 13:01:03.282862 8868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a303b287ecc4822bd13e022f58cdb82. Latest consensus state: current_term: 2 leader_uuid: "7a303b287ecc4822bd13e022f58cdb82" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:03.283152 8868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:01:03.285138 8868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7a303b287ecc4822bd13e022f58cdb82" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:03.285483 8868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14031fd252a44b91b756b343b29303c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:01:03.286453 8867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7a303b287ecc4822bd13e022f58cdb82" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9933c1c7f5644e159c996c2efbaabcb6" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33027 } } peers { permanent_uuid: "14031fd252a44b91b756b343b29303c9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 43867 } attrs { promote: false } } peers { permanent_uuid: "7a303b287ecc4822bd13e022f58cdb82" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 39717 } attrs { promote: false } } }
I20260619 13:01:03.286793 8867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:03.301800 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.294318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42863 (local address 127.0.0.1:53004)
0619 13:01:03.294543 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.294563 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:03.294667 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:03.295257 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:03.295261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:03.295288 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:03.295519 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.295528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.297487 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.297492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.299457 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.299475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.300993 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.300998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.301190 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.301232 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 13:01:03.301297 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 13:01:03.301426 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 13:01:03.303141 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.294452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.1.87.65:42863)
0619 13:01:03.294924 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:03.294931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:03.294950 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:01:03.295023 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:03.295028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:03.295098 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:03.295649 (+ 551us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:03.295793 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.297340 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.299694 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.300846 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.302470 (+ 1624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.302579 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:03.302627 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 13:01:03.302803 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":284,"threads_started":1}
I20260619 13:01:03.304236 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.295111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:43129 (local address 127.0.0.1:35264)
0619 13:01:03.295402 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.295418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:03.295511 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:03.296494 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:03.296499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:03.296525 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:03.296835 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.296848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.298781 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.298786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.300820 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.300836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.303447 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.303453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.303643 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.303680 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:01:03.303746 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 13:01:03.303899 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260619 13:01:03.304998 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.295670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.1.87.66:43129)
0619 13:01:03.296166 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:03.296171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:03.296189 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:01:03.296260 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:03.296265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:03.296326 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:03.297472 (+ 1146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:03.297625 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.298625 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.301925 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.303198 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.304532 (+ 1334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.304598 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:03.304634 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 13:01:03.304776 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":263,"threads_started":1}
I20260619 13:01:03.306274 8588 tablet_service.cc:1467] Processing CreateTablet for tablet 7e217479ad944b47b455711098bed503 (DEFAULT_TABLE table=test-table [id=228470f357af401b92dda649ede7f46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:03.306694 8588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e217479ad944b47b455711098bed503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:03.306614 8443 tablet_service.cc:1467] Processing CreateTablet for tablet 7e217479ad944b47b455711098bed503 (DEFAULT_TABLE table=test-table [id=228470f357af401b92dda649ede7f46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:03.306990 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e217479ad944b47b455711098bed503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:03.311272 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.295117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36091 (local address 127.0.0.1:42022)
0619 13:01:03.301999 (+ 6882us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.302015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:03.302118 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:03.305725 (+ 3607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:03.305729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:03.305752 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:03.305888 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.305897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.307191 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.307198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.308987 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:03.309000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.310450 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.310458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:03.310702 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.310735 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:01:03.310826 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 13:01:03.310951 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6802,"mutex_wait_us":30,"thread_start_us":10679,"threads_started":1}
I20260619 13:01:03.311291 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.295373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.1.87.67:36091)
0619 13:01:03.305402 (+ 10029us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:03.305409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:03.305431 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:01:03.305501 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:03.305508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:03.305579 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:03.305714 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:03.306154 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.307080 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.309157 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.310308 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.310435 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.310533 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:03.310871 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 13:01:03.311008 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9884,"thread_start_us":9788,"threads_started":1}
I20260619 13:01:03.312335 8729 tablet_service.cc:1467] Processing CreateTablet for tablet 7e217479ad944b47b455711098bed503 (DEFAULT_TABLE table=test-table [id=228470f357af401b92dda649ede7f46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:03.312527 8890 tablet_bootstrap.cc:492] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: Bootstrap starting.
I20260619 13:01:03.312722 8729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e217479ad944b47b455711098bed503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:03.313392 8891 tablet_bootstrap.cc:492] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: Bootstrap starting.
I20260619 13:01:03.314342 8890 tablet_bootstrap.cc:654] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:03.314980 8891 tablet_bootstrap.cc:654] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:03.316689 8890 tablet_bootstrap.cc:492] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: No bootstrap required, opened a new log
I20260619 13:01:03.316828 8890 ts_tablet_manager.cc:1397] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:01:03.317051 8891 tablet_bootstrap.cc:492] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: No bootstrap required, opened a new log
I20260619 13:01:03.317194 8891 ts_tablet_manager.cc:1397] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:01:03.317545 8890 raft_consensus.cc:348] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } }
I20260619 13:01:03.317734 8890 raft_consensus.cc:374] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:03.317827 8890 raft_consensus.cc:729] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b9fc73b1714ddabd5d5cd927adacf1, State: Initialized, Role: FOLLOWER
I20260619 13:01:03.317904 8891 raft_consensus.cc:348] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } }
I20260619 13:01:03.318137 8891 raft_consensus.cc:374] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:03.318068 8890 consensus_queue.cc:260] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } }
I20260619 13:01:03.318223 8891 raft_consensus.cc:729] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259e01b38a444737934f88d0c1ccad01, State: Initialized, Role: FOLLOWER
I20260619 13:01:03.318445 8891 consensus_queue.cc:260] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } }
I20260619 13:01:03.318892 8890 ts_tablet_manager.cc:1428] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:01:03.322593 8509 heartbeater.cc:503] Master 127.1.87.124:39717 was elected leader, sending a full tablet report...
I20260619 13:01:03.324817 8896 tablet_bootstrap.cc:492] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: Bootstrap starting.
I20260619 13:01:03.325825 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.319123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.65:51915)
0619 13:01:03.325442 (+ 6319us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.325581 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1199,"thread_start_us":1094,"threads_started":1}
I20260619 13:01:03.326808 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.326483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.65:34629)
0619 13:01:03.326647 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.326718 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 13:01:03.327859 8896 tablet_bootstrap.cc:654] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:03.327924 8891 ts_tablet_manager.cc:1428] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260619 13:01:03.329098 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.328070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.66:51855)
0619 13:01:03.328676 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.328816 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":378,"threads_started":1}
I20260619 13:01:03.330430 8896 tablet_bootstrap.cc:492] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: No bootstrap required, opened a new log
I20260619 13:01:03.330587 8896 ts_tablet_manager.cc:1397] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 13:01:03.331214 8896 raft_consensus.cc:348] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } }
I20260619 13:01:03.331498 8896 raft_consensus.cc:374] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:03.331630 8896 raft_consensus.cc:729] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1707fe2ccdf4c9fbb06aa52a092446b, State: Initialized, Role: FOLLOWER
I20260619 13:01:03.331847 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.329685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.66:48933)
0619 13:01:03.331584 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.331677 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1800,"threads_started":1}
I20260619 13:01:03.331959 8896 consensus_queue.cc:260] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } }
I20260619 13:01:03.332607 8654 heartbeater.cc:503] Master 127.1.87.124:39717 was elected leader, sending a full tablet report...
I20260619 13:01:03.333133 8896 ts_tablet_manager.cc:1428] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 13:01:03.333880 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.333099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.67:33483)
0619 13:01:03.333616 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.333716 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"threads_started":1}
I20260619 13:01:03.335439 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.334983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.67:46469)
0619 13:01:03.335184 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.335271 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 13:01:03.335920 8795 heartbeater.cc:503] Master 127.1.87.124:39717 was elected leader, sending a full tablet report...
I20260619 13:01:03.407513 8901 raft_consensus.cc:482] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:03.407752 8901 raft_consensus.cc:504] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } }
I20260619 13:01:03.408533 8901 leader_election.cc:288] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863), 259e01b38a444737934f88d0c1ccad01 (127.1.87.66:43129)
I20260619 13:01:03.409220 8608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e217479ad944b47b455711098bed503" candidate_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259e01b38a444737934f88d0c1ccad01" is_pre_election: true
I20260619 13:01:03.409219 8463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e217479ad944b47b455711098bed503" candidate_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" is_pre_election: true
I20260619 13:01:03.409523 8463 raft_consensus.cc:2454] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1707fe2ccdf4c9fbb06aa52a092446b in term 0.
I20260619 13:01:03.409528 8608 raft_consensus.cc:2454] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1707fe2ccdf4c9fbb06aa52a092446b in term 0.
I20260619 13:01:03.410041 8685 leader_election.cc:302] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259e01b38a444737934f88d0c1ccad01, f1707fe2ccdf4c9fbb06aa52a092446b; no voters:
I20260619 13:01:03.410353 8901 raft_consensus.cc:2781] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:01:03.410468 8901 raft_consensus.cc:482] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:01:03.410591 8901 raft_consensus.cc:3037] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:03.412746 8901 raft_consensus.cc:504] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } }
I20260619 13:01:03.413280 8901 leader_election.cc:288] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [CANDIDATE]: Term 1 election: Requested vote from peers a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863), 259e01b38a444737934f88d0c1ccad01 (127.1.87.66:43129)
I20260619 13:01:03.413833 8608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e217479ad944b47b455711098bed503" candidate_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259e01b38a444737934f88d0c1ccad01"
I20260619 13:01:03.413833 8463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e217479ad944b47b455711098bed503" candidate_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1"
I20260619 13:01:03.414052 8608 raft_consensus.cc:3037] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:03.414050 8463 raft_consensus.cc:3037] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:03.417060 8463 raft_consensus.cc:2454] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1707fe2ccdf4c9fbb06aa52a092446b in term 1.
I20260619 13:01:03.417060 8608 raft_consensus.cc:2454] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1707fe2ccdf4c9fbb06aa52a092446b in term 1.
I20260619 13:01:03.417533 8682 leader_election.cc:302] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1b9fc73b1714ddabd5d5cd927adacf1, f1707fe2ccdf4c9fbb06aa52a092446b; no voters:
I20260619 13:01:03.417805 8901 raft_consensus.cc:2781] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:03.418330 8901 raft_consensus.cc:686] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 LEADER]: Becoming Leader. State: Replica: f1707fe2ccdf4c9fbb06aa52a092446b, State: Running, Role: LEADER
I20260619 13:01:03.418802 8901 consensus_queue.cc:237] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } }
I20260619 13:01:03.419454 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.418903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.67:37607)
0619 13:01:03.419092 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.419189 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 13:01:03.422415 8330 catalog_manager.cc:5495] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b reported cstate change: term changed from 0 to 1, leader changed from <none> to f1707fe2ccdf4c9fbb06aa52a092446b (127.1.87.67). New cstate: current_term: 1 leader_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1707fe2ccdf4c9fbb06aa52a092446b" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 } health_report { overall_health: UNKNOWN } } }
I20260619 13:01:03.458683 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.454295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.1.87.67:36091)
0619 13:01:03.454480 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:03.454484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:03.454500 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:01:03.454581 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:03.454585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:03.454657 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:03.454787 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:01:03.455072 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.455811 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.456957 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.457343 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.457431 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.457756 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:01:03.458262 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:01:03.458363 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:03.458435 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 13:01:03.458495 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:01:03.462682 8463 raft_consensus.cc:1261] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [term 1 FOLLOWER]: Refusing update from remote peer f1707fe2ccdf4c9fbb06aa52a092446b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:03.463043 8608 raft_consensus.cc:1261] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [term 1 FOLLOWER]: Refusing update from remote peer f1707fe2ccdf4c9fbb06aa52a092446b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:03.463452 8903 consensus_queue.cc:1035] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1b9fc73b1714ddabd5d5cd927adacf1" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:03.463835 8901 consensus_queue.cc:1035] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [LEADER]: Connected to new peer: Peer: permanent_uuid: "259e01b38a444737934f88d0c1ccad01" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:03.466534 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.466018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.65:42447)
0619 13:01:03.466188 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.466312 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 13:01:03.467825 8797 tablet.cc:2369] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:01:03.470322 8906 mvcc.cc:205] Tried to move back new op lower bound from 7298556163936632832 to 7298556163767611392. Current Snapshot: MvccSnapshot[applied={T|T < 7298556163936632832}]
I20260619 13:01:03.470606 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.466871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.66:53805)
0619 13:01:03.470217 (+ 3346us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.470363 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2662}
I20260619 13:01:03.475134 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.474384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.67:57771)
0619 13:01:03.474904 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.474987 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
I20260619 13:01:03.481142 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.480492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.65:60267)
0619 13:01:03.480803 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.480924 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260619 13:01:03.491487 8329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:03.491844 8329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:03.498967 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.498337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.67:48493)
0619 13:01:03.498639 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.498735 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 13:01:03.501753 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.501238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.1.87.66:37567)
0619 13:01:03.501478 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.501570 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 13:01:03.503894 8329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a303b287ecc4822bd13e022f58cdb82: Sending DeleteTablet for 3 replicas of tablet 7e217479ad944b47b455711098bed503
I20260619 13:01:03.505366 8588 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e217479ad944b47b455711098bed503 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:03 UTC) from {username='oryx', principal='oryx/127.1.87.124@KRBTEST.COM'} at 127.0.0.1:35264
I20260619 13:01:03.507622 8443 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e217479ad944b47b455711098bed503 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:03 UTC) from {username='oryx', principal='oryx/127.1.87.124@KRBTEST.COM'} at 127.0.0.1:53004
I20260619 13:01:03.507665 8729 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e217479ad944b47b455711098bed503 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:03 UTC) from {username='oryx', principal='oryx/127.1.87.124@KRBTEST.COM'} at 127.0.0.1:42022
I20260619 13:01:03.508116 8442 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e217479ad944b47b455711098bed503 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:03 UTC) from {username='oryx', principal='oryx/127.1.87.124@KRBTEST.COM'} at 127.0.0.1:53004
I20260619 13:01:03.508340 8916 tablet_replica.cc:330] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: stopping tablet replica
I20260619 13:01:03.508524 8916 raft_consensus.cc:2229] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:03.508616 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 8302
I20260619 13:01:03.508747 8916 raft_consensus.cc:2258] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:03.518499 8916 ts_tablet_manager.cc:1905] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:03.519164 8918 tablet_replica.cc:330] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: stopping tablet replica
I20260619 13:01:03.519390 8918 raft_consensus.cc:2229] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:01:03.519870 8917 tablet_replica.cc:330] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: stopping tablet replica
I20260619 13:01:03.519981 8918 raft_consensus.cc:2258] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:03.520063 8917 raft_consensus.cc:2229] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:03.520445 8917 raft_consensus.cc:2258] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:03.521191 8918 ts_tablet_manager.cc:1905] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:03.523728 8917 ts_tablet_manager.cc:1905] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:03.528688 8918 ts_tablet_manager.cc:1918] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:03.528873 8918 log.cc:1196] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/7e217479ad944b47b455711098bed503
I20260619 13:01:03.529426 8918 ts_tablet_manager.cc:1939] T 7e217479ad944b47b455711098bed503 P f1707fe2ccdf4c9fbb06aa52a092446b: Deleting consensus metadata
I20260619 13:01:03.530814 8314 catalog_manager.cc:4841] TS f1707fe2ccdf4c9fbb06aa52a092446b (127.1.87.67:36091): tablet 7e217479ad944b47b455711098bed503 (table test-table [id=228470f357af401b92dda649ede7f46d]) successfully deleted
I20260619 13:01:03.534607 8917 ts_tablet_manager.cc:1918] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:03.534756 8917 log.cc:1196] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/7e217479ad944b47b455711098bed503
I20260619 13:01:03.535233 8917 ts_tablet_manager.cc:1939] T 7e217479ad944b47b455711098bed503 P a1b9fc73b1714ddabd5d5cd927adacf1: Deleting consensus metadata
I20260619 13:01:03.536437 8314 catalog_manager.cc:4841] TS a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863): tablet 7e217479ad944b47b455711098bed503 (table test-table [id=228470f357af401b92dda649ede7f46d]) successfully deleted
I20260619 13:01:03.536770 8916 ts_tablet_manager.cc:1918] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:03.536971 8916 log.cc:1196] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/7e217479ad944b47b455711098bed503
W20260619 13:01:03.537076 8314 catalog_manager.cc:4808] TS a1b9fc73b1714ddabd5d5cd927adacf1 (127.1.87.65:42863): delete failed for tablet 7e217479ad944b47b455711098bed503 because the tablet was not found. No further retry: Not found: Tablet not found: 7e217479ad944b47b455711098bed503
I20260619 13:01:03.537482 8916 ts_tablet_manager.cc:1939] T 7e217479ad944b47b455711098bed503 P 259e01b38a444737934f88d0c1ccad01: Deleting consensus metadata
I20260619 13:01:03.538401 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.510431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.1.87.65:42863)
0619 13:01:03.510666 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:03.510674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:03.519534 (+ 8860us) server_negotiation.cc:407] Connection header received
0619 13:01:03.519827 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:03.519836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:03.519926 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:03.520006 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:03.522115 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.523144 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.524930 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.525621 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.525720 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.529315 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:03.529334 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:03.529354 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:03.529405 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:03.533803 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:03.534771 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:03.534784 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:03.534791 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:03.534982 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:03.535726 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:03.535732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:03.535736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:03.536944 (+ 1208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:03.537228 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:03.537824 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 13:01:03.538096 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:01:03.538612 8317 catalog_manager.cc:4841] TS 259e01b38a444737934f88d0c1ccad01 (127.1.87.66:43129): tablet 7e217479ad944b47b455711098bed503 (table test-table [id=228470f357af401b92dda649ede7f46d]) successfully deleted
I20260619 13:01:03.539225 8503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:03.649906 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 8302
I20260619 13:01:03.672789 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 8519
I20260619 13:01:03.676767 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.676353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54404)
0619 13:01:03.676505 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.676597 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 13:01:03.677922 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.677514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33027 (local address 127.0.0.1:54412)
0619 13:01:03.677683 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:03.677777 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 13:01:03.688952 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.673597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.1.87.66:43129)
0619 13:01:03.673810 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:03.673818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:03.673925 (+ 107us) server_negotiation.cc:407] Connection header received
0619 13:01:03.674130 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:03.674137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:03.674239 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:03.674361 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:03.675781 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.676871 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.678308 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.678962 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.679120 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.681746 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:03.681764 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:03.681786 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:03.681854 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:03.685578 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:03.686466 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:03.686476 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:03.686481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:03.686644 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:03.687174 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:03.687180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:03.687183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:03.687740 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:03.688049 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:03.688385 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 13:01:03.688593 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 13:01:03.689646 8648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:03.807228 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 8519
I20260619 13:01:03.829504 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 8666
I20260619 13:01:03.843822 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:03.830285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.1.87.67:36091)
0619 13:01:03.830454 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:03.830461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:03.830486 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:01:03.830709 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:03.830717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:03.830812 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:03.830935 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:03.832307 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.833065 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.834402 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:03.834851 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:03.835001 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:03.837508 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:03.837530 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:03.837545 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:03.837583 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:03.840831 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:03.841644 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:03.841653 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:03.841661 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:03.841819 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:03.842199 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:03.842205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:03.842209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:03.842567 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:03.842890 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:03.843180 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 13:01:03.843429 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:01:03.844728 8789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:03.969224 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 8666
I20260619 13:01:03.990515 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 7998
I20260619 13:01:03.991680 8164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:04.106362 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 7998
I20260619 13:01:04.126720 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 8098
I20260619 13:01:04.127780 8360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:04.244091 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 8098
2026-06-19T13:01:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9190 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:01:04 dist-test-slave-tpsm krb5kdc[8927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:01:04 dist-test-slave-tpsm krb5kdc[8927](info): set up 2 sockets
Jun 19 13:01:04 dist-test-slave-tpsm krb5kdc[8927](info): commencing operation
krb5kdc: starting...
W20260619 13:01:05.340260 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:01:05 dist-test-slave-tpsm krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:01:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:01:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.126.keytab.
Entry for principal oryx/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.126.keytab.
I20260619 13:01:05.555042 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:45629
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.126.keytab
--principal=oryx/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:45629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:05.868860 8940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:05.869210 8940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:05.869499 8940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:05.877074 8940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:01:05.877225 8940 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 13:01:05.877326 8940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:05.877452 8940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:05.877543 8940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:01:05.877641 8940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:01:05.887596 8940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:45629
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.126.keytab
--principal=oryx/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:45629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:05.888891 8940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:05.890218 8940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:05.901775 8945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:05.901820 8948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:05.901859 8946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:05.902709 8940 server_base.cc:1034] running on GCE node
I20260619 13:01:05.903486 8940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:05.904783 8940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:05.906021 8940 hybrid_clock.cc:648] HybridClock initialized: now 1781874065905975 us; error 40 us; skew 500 ppm
I20260619 13:01:05.906432 8940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:05 dist-test-slave-tpsm krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874065, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:05.909396 8940 init.cc:377] Logged in from keytab as oryx/127.1.87.126@KRBTEST.COM (short username oryx)
I20260619 13:01:05.910791 8940 webserver.cc:460] Webserver started at http://127.1.87.126:37697/ using document root <none> and password file <none>
I20260619 13:01:05.911513 8940 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:05.911638 8940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:05.911952 8940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:05.914692 8940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "9028c1ef37e042e1801476ab421d0f78"
format_stamp: "Formatted at 2026-06-19 13:01:05 on dist-test-slave-tpsm"
server_key: "fab62c86eb845a825681f638a40c4048"
server_key_iv: "49cf753a07089049514398f72944dbd7"
server_key_version: "encryptionkey@0"
I20260619 13:01:05.915513 8940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "9028c1ef37e042e1801476ab421d0f78"
format_stamp: "Formatted at 2026-06-19 13:01:05 on dist-test-slave-tpsm"
server_key: "fab62c86eb845a825681f638a40c4048"
server_key_iv: "49cf753a07089049514398f72944dbd7"
server_key_version: "encryptionkey@0"
I20260619 13:01:05.920255 8940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:01:05.923758 8955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:05.925130 8940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:01:05.925323 8940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "9028c1ef37e042e1801476ab421d0f78"
format_stamp: "Formatted at 2026-06-19 13:01:05 on dist-test-slave-tpsm"
server_key: "fab62c86eb845a825681f638a40c4048"
server_key_iv: "49cf753a07089049514398f72944dbd7"
server_key_version: "encryptionkey@0"
I20260619 13:01:05.925575 8940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:05.953848 8940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:05.958257 8940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:05.958590 8940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:05.982054 8940 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:45629
I20260619 13:01:05.982072 9016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:45629 every 8 connection(s)
I20260619 13:01:05.983592 8940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:01:05.988056 9017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:05.994064 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 8940
I20260619 13:01:05.994307 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:01:05.995153 1373 external_mini_cluster.cc:1428] Setting key d09c06acc1ae70a87cabdc128e266a62
I20260619 13:01:05.999773 9017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78: Bootstrap starting.
I20260619 13:01:06.003422 9017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:06.004520 9017 log.cc:824] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:06.007632 9017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78: No bootstrap required, opened a new log
Jun 19 13:01:06 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:01:06.015897 9017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9028c1ef37e042e1801476ab421d0f78" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45629 } }
I20260619 13:01:06.016299 9017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:06.016424 9017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9028c1ef37e042e1801476ab421d0f78, State: Initialized, Role: FOLLOWER
I20260619 13:01:06.017068 9017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9028c1ef37e042e1801476ab421d0f78" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45629 } }
I20260619 13:01:06.017329 9017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:01:06.017481 9017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:01:06.017668 9017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:06.018891 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:05.997794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.1.87.126:45629)
0619 13:01:05.998589 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:05.998601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:05.998642 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:01:06.000454 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:06.000477 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:06.000852 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:06.001169 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:06.002862 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.004299 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.005896 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.006615 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.006762 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:06.011455 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:06.011465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:06.011480 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:06.011520 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:06.014669 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:06.015414 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:06.015422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:06.015429 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:06.015599 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:06.015978 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:06.015984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:06.015987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:06.016681 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:06.016982 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:06.017412 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 13:01:06.017875 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":385,"threads_started":1}
I20260619 13:01:06.020180 9017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9028c1ef37e042e1801476ab421d0f78" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45629 } }
I20260619 13:01:06.020699 9017 leader_election.cc:302] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9028c1ef37e042e1801476ab421d0f78; no voters:
I20260619 13:01:06.021349 9017 leader_election.cc:288] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:01:06.021478 9022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:06.022333 9022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [term 1 LEADER]: Becoming Leader. State: Replica: 9028c1ef37e042e1801476ab421d0f78, State: Running, Role: LEADER
I20260619 13:01:06.022851 9022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9028c1ef37e042e1801476ab421d0f78" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45629 } }
I20260619 13:01:06.023600 9017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:01:06.026022 9024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9028c1ef37e042e1801476ab421d0f78. Latest consensus state: current_term: 1 leader_uuid: "9028c1ef37e042e1801476ab421d0f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9028c1ef37e042e1801476ab421d0f78" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45629 } } }
I20260619 13:01:06.026496 9024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:06.027138 9023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9028c1ef37e042e1801476ab421d0f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9028c1ef37e042e1801476ab421d0f78" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45629 } } }
I20260619 13:01:06.027427 9023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:06.028296 9031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:01:06.034716 9031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:01:06.043680 9031 catalog_manager.cc:1347] Generated new cluster ID: b239a229713848709527754207757b65
I20260619 13:01:06.043802 9031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:01:06.066738 9031 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:01:06.068547 9031 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:01:06.078841 9031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78: Generated new TSK 0
I20260619 13:01:06.079927 9031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab.
Entry for principal oryx/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab.
I20260619 13:01:06.130511 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45629
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab
--principal=oryx/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:06.450147 9043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:06.450407 9043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:06.450570 9043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:06.457845 9043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:06.457955 9043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:06.458141 9043 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:01:06.458269 9043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:01:06.469683 9043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab
--principal=oryx/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.87.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:06.471423 9043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:06.473412 9043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:06.486140 9048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:06.486058 9049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:06.486091 9051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:06.486934 9043 server_base.cc:1034] running on GCE node
I20260619 13:01:06.487630 9043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:06.488513 9043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:06.489766 9043 hybrid_clock.cc:648] HybridClock initialized: now 1781874066489711 us; error 52 us; skew 500 ppm
I20260619 13:01:06.490072 9043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:06 dist-test-slave-tpsm krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874066, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:06.493160 9043 init.cc:377] Logged in from keytab as oryx/127.1.87.65@KRBTEST.COM (short username oryx)
I20260619 13:01:06.494585 9043 webserver.cc:460] Webserver started at http://127.1.87.65:39601/ using document root <none> and password file <none>
I20260619 13:01:06.495198 9043 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:06.495277 9043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:06.495643 9043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:06.498349 9043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "a0b28a41b03e41ee864e404b5c45f67e"
format_stamp: "Formatted at 2026-06-19 13:01:06 on dist-test-slave-tpsm"
server_key: "a5ce07a4e2b5cf64ca76d7a9c1480a87"
server_key_iv: "9a51b087eafdfab4162b89497476b0ce"
server_key_version: "encryptionkey@0"
I20260619 13:01:06.499044 9043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "a0b28a41b03e41ee864e404b5c45f67e"
format_stamp: "Formatted at 2026-06-19 13:01:06 on dist-test-slave-tpsm"
server_key: "a5ce07a4e2b5cf64ca76d7a9c1480a87"
server_key_iv: "9a51b087eafdfab4162b89497476b0ce"
server_key_version: "encryptionkey@0"
I20260619 13:01:06.503947 9043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 13:01:06.507103 9058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:06.508404 9043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:01:06.508522 9043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "a0b28a41b03e41ee864e404b5c45f67e"
format_stamp: "Formatted at 2026-06-19 13:01:06 on dist-test-slave-tpsm"
server_key: "a5ce07a4e2b5cf64ca76d7a9c1480a87"
server_key_iv: "9a51b087eafdfab4162b89497476b0ce"
server_key_version: "encryptionkey@0"
I20260619 13:01:06.508697 9043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:06.528348 9043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:06.532850 9043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:06.533072 9043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:06.535661 9043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:06.535751 9043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:06.535905 9043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:06.535975 9043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:01:06 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874066, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:01:06.559442 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:06.536523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:36231 (local address 127.1.87.126:45629)
0619 13:01:06.536973 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:06.536978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:06.538091 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 13:01:06.540252 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:06.540258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:06.540351 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:06.540472 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:06.542564 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.543877 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.546764 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.547230 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.547422 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:06.552701 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:06.552708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:06.552712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:06.552763 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:06.556092 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:06.556918 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:06.556926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:06.556931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:06.557063 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:06.557504 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:06.557511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:06.557514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:06.557787 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:06.557934 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:06.558832 (+ 898us) server_negotiation.cc:299] Negotiation successful
0619 13:01:06.559042 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":263,"threads_started":1}
I20260619 13:01:06.560604 9068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:06.537137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45629 (local address 127.1.87.65:36231)
0619 13:01:06.537945 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:06.537984 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:06.539961 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:06.540704 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:06.540719 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:06.541470 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:06.542331 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:06.542358 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.544057 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.544065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:06.546581 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:06.546593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.547534 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.547582 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:06.548032 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:06.548918 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:06.548943 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:06.552191 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:06.556259 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:06.556268 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:06.556284 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:06.556793 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:06.557183 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:06.557188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:06.557191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:06.557392 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:06.557927 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:06.557934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:06.558382 (+ 448us) client_negotiation.cc:769] Sending connection context
0619 13:01:06.558771 (+ 389us) client_negotiation.cc:240] Negotiation successful
0619 13:01:06.559295 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":340,"threads_started":1}
I20260619 13:01:06.580375 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:06.567021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:57281 (local address 127.1.87.126:45629)
0619 13:01:06.567173 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:06.567177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:06.567794 (+ 617us) server_negotiation.cc:407] Connection header received
0619 13:01:06.567980 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:06.567986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:06.568056 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:06.568157 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:06.569008 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.569735 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.570981 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.571548 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.571703 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:06.574024 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:06.574029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:06.574033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:06.574060 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:06.577640 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:06.578390 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:06.578396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:06.578400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:06.578536 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:06.578915 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:06.578919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:06.578923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:06.579184 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:06.579374 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:06.579808 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 13:01:06.580025 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:01:06.580928 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:06.567053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45629 (local address 127.1.87.65:57281)
0619 13:01:06.567691 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:06.567713 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:06.567862 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:06.568177 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:06.568182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:06.568613 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:06.568878 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:06.568886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.569866 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.569871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:06.570842 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:06.570853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:06.571755 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:06.571762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:06.571981 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:06.572668 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:06.572684 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:06.573820 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:06.577774 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:06.577780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:06.577784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:06.578274 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:06.578657 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:06.578663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:06.578669 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:06.578801 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:06.579303 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:06.579308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:06.579562 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 13:01:06.579737 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 13:01:06.580053 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":396,"threads_started":1}
I20260619 13:01:06.591612 9043 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:38991
I20260619 13:01:06.591635 9178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:38991 every 8 connection(s)
I20260619 13:01:06.592767 9043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:01:06.594687 9179 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45629
I20260619 13:01:06.594920 9179 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:06.595607 9179 heartbeater.cc:511] Master 127.1.87.126:45629 requested a full tablet report, sending...
I20260619 13:01:06.597486 8972 ts_manager.cc:194] Registered new tserver with Master: a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991)
I20260619 13:01:06.599466 8972 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:36231
I20260619 13:01:06.601732 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 9043
I20260619 13:01:06.601888 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:01:06.602458 1373 external_mini_cluster.cc:1428] Setting key 8fe42d8ec89fe54ee05cfd83eb6220ad
WARNING: no policy specified for oryx/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab.
Entry for principal oryx/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab.
I20260619 13:01:06.646275 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45629
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab
--principal=oryx/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:06.968082 9183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:06.968369 9183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:06.968564 9183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:06.975765 9183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:06.975875 9183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:06.976083 9183 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:01:06.976213 9183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:01:06.989014 9183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab
--principal=oryx/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.87.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:06.990549 9183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:06.992291 9183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:07.007148 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:07.007118 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:07.007184 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:07.007980 9183 server_base.cc:1034] running on GCE node
I20260619 13:01:07.008607 9183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:07.009443 9183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:07.010641 9183 hybrid_clock.cc:648] HybridClock initialized: now 1781874067010589 us; error 29 us; skew 500 ppm
I20260619 13:01:07.010872 9183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:07 dist-test-slave-tpsm krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874067, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:07.018114 9183 init.cc:377] Logged in from keytab as oryx/127.1.87.66@KRBTEST.COM (short username oryx)
I20260619 13:01:07.019531 9183 webserver.cc:460] Webserver started at http://127.1.87.66:35329/ using document root <none> and password file <none>
I20260619 13:01:07.020147 9183 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:07.020227 9183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:07.020466 9183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:07.023085 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "aaf31521f80646bbadede2ce9289bb11"
format_stamp: "Formatted at 2026-06-19 13:01:07 on dist-test-slave-tpsm"
server_key: "bc18c4c2cce8a9fdf58951a3642c527d"
server_key_iv: "ffeb3360ce109d0f44a8d056ce179c7f"
server_key_version: "encryptionkey@0"
I20260619 13:01:07.023916 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "aaf31521f80646bbadede2ce9289bb11"
format_stamp: "Formatted at 2026-06-19 13:01:07 on dist-test-slave-tpsm"
server_key: "bc18c4c2cce8a9fdf58951a3642c527d"
server_key_iv: "ffeb3360ce109d0f44a8d056ce179c7f"
server_key_version: "encryptionkey@0"
I20260619 13:01:07.029091 9183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:01:07.032500 9198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:07.033772 9183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:01:07.033932 9183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "aaf31521f80646bbadede2ce9289bb11"
format_stamp: "Formatted at 2026-06-19 13:01:07 on dist-test-slave-tpsm"
server_key: "bc18c4c2cce8a9fdf58951a3642c527d"
server_key_iv: "ffeb3360ce109d0f44a8d056ce179c7f"
server_key_version: "encryptionkey@0"
I20260619 13:01:07.034143 9183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:07.054840 9183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:07.059152 9183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:07.059494 9183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:07.062053 9183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:07.062163 9183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:07.062288 9183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:07.062358 9183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:01:07 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874067, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:01:07.084794 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.062720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:57503 (local address 127.1.87.126:45629)
0619 13:01:07.062904 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:07.062913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:07.064464 (+ 1551us) server_negotiation.cc:407] Connection header received
0619 13:01:07.066379 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:07.066385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:07.066518 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:07.066639 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:07.068674 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.070011 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.072575 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.072995 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.073139 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.077861 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:07.077867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:07.077870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:07.077901 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:07.081213 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.082124 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.082128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.082130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.082299 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.082819 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.082824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.082827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.083132 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:07.083366 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:07.084394 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0619 13:01:07.084564 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 13:01:07.086573 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.063468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45629 (local address 127.1.87.66:57503)
0619 13:01:07.064311 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:07.064354 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:07.066113 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:07.066854 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:07.066865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:07.067605 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:07.068466 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.068495 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.070178 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.070189 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.072390 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.072404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.073238 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.073247 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.073552 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.074372 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:07.074424 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:07.077628 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:07.081382 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:07.081391 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:07.081411 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:07.081984 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:07.082445 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:07.082452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:07.082457 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:07.082690 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:07.083304 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:07.083347 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:07.083926 (+ 579us) client_negotiation.cc:769] Sending connection context
0619 13:01:07.085023 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0619 13:01:07.085386 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":360,"threads_started":1}
I20260619 13:01:07.109972 9183 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:38225
I20260619 13:01:07.109989 9317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:38225 every 8 connection(s)
I20260619 13:01:07.111263 9183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:01:07.113451 9318 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45629
I20260619 13:01:07.113685 9318 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:07.114385 9318 heartbeater.cc:511] Master 127.1.87.126:45629 requested a full tablet report, sending...
I20260619 13:01:07.115908 8972 ts_manager.cc:194] Registered new tserver with Master: aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225)
I20260619 13:01:07.116677 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 9183
I20260619 13:01:07.116889 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:01:07.117017 8972 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:57503
I20260619 13:01:07.117463 9295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.100886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45629 (local address 127.1.87.66:45659)
0619 13:01:07.101912 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:07.101937 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:07.102131 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:07.102500 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:07.102505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:07.102951 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:07.103229 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.103237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.104597 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.104604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.106919 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.106938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.108086 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.108093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.108235 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.108900 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:07.108933 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:07.110286 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:07.113891 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:07.113900 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:07.113907 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:07.114638 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:07.115070 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:07.115078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:07.115084 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:07.115280 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:07.116189 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:07.116194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:07.116379 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 13:01:07.116515 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 13:01:07.116868 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":600,"threads_started":1}
I20260619 13:01:07.117569 1373 external_mini_cluster.cc:1428] Setting key 9632eee8e6c283d7dfa37b894e067857
I20260619 13:01:07.119024 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.100887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:45659 (local address 127.1.87.126:45629)
0619 13:01:07.101070 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:07.101076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:07.102044 (+ 968us) server_negotiation.cc:407] Connection header received
0619 13:01:07.102276 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:07.102280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:07.102352 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:07.102464 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:07.103388 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.104432 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.107092 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.107866 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.108013 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.110454 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:07.110458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:07.110462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:07.110489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:07.113769 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.114781 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.114786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.114789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.114932 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.115462 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.115470 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.115480 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.116036 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:07.118311 (+ 2275us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:07.118466 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 13:01:07.118714 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for oryx/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab.
Entry for principal oryx/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab.
I20260619 13:01:07.162249 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45629
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab
--principal=oryx/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:07.486526 9322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:07.486879 9322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:07.487071 9322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:07.494993 9322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:07.495185 9322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:07.495426 9322 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:01:07.495594 9322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:01:07.505973 9322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab
--principal=oryx/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.87.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:07.507591 9322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:07.508983 9322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:07.525583 9327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:07.526347 9328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:07.527084 9322 server_base.cc:1034] running on GCE node
W20260619 13:01:07.527539 9330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:07.528234 9322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:07.529498 9322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:07.530789 9322 hybrid_clock.cc:648] HybridClock initialized: now 1781874067530708 us; error 60 us; skew 500 ppm
I20260619 13:01:07.531133 9322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:07 dist-test-slave-tpsm krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874067, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:07.534485 9322 init.cc:377] Logged in from keytab as oryx/127.1.87.67@KRBTEST.COM (short username oryx)
I20260619 13:01:07.536185 9322 webserver.cc:460] Webserver started at http://127.1.87.67:42907/ using document root <none> and password file <none>
I20260619 13:01:07.536901 9322 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:07.537034 9322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:07.537407 9322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:07.540481 9322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "261db81c10d545a99baeb865951226ac"
format_stamp: "Formatted at 2026-06-19 13:01:07 on dist-test-slave-tpsm"
server_key: "c7567e3a7dc4be81d25e602bca62c0fa"
server_key_iv: "b0a3123f944e7559b12447ea217fee9c"
server_key_version: "encryptionkey@0"
I20260619 13:01:07.541272 9322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "261db81c10d545a99baeb865951226ac"
format_stamp: "Formatted at 2026-06-19 13:01:07 on dist-test-slave-tpsm"
server_key: "c7567e3a7dc4be81d25e602bca62c0fa"
server_key_iv: "b0a3123f944e7559b12447ea217fee9c"
server_key_version: "encryptionkey@0"
I20260619 13:01:07.547228 9322 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 13:01:07.551097 9337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:07.552603 9322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:01:07.552771 9322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "261db81c10d545a99baeb865951226ac"
format_stamp: "Formatted at 2026-06-19 13:01:07 on dist-test-slave-tpsm"
server_key: "c7567e3a7dc4be81d25e602bca62c0fa"
server_key_iv: "b0a3123f944e7559b12447ea217fee9c"
server_key_version: "encryptionkey@0"
I20260619 13:01:07.553013 9322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:07.580459 9322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:07.584698 9322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:07.584991 9322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:07.587756 9322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:07.587863 9322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:07.587989 9322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:07.588074 9322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:07.601953 9179 heartbeater.cc:503] Master 127.1.87.126:45629 was elected leader, sending a full tablet report...
Jun 19 13:01:07 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874067, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:01:07.612818 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.588382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:39299 (local address 127.1.87.126:45629)
0619 13:01:07.588540 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:07.588544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:07.590378 (+ 1834us) server_negotiation.cc:407] Connection header received
0619 13:01:07.593225 (+ 2847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:07.593232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:07.593330 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:07.593455 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:07.595634 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.596728 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.599159 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.599833 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.599985 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.604993 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:07.604999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:07.605002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:07.605032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:07.608949 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.609740 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.609748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.609754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.609928 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.610609 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.610616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.610621 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.611013 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:07.611185 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:07.612233 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0619 13:01:07.612464 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 13:01:07.613874 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.589241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45629 (local address 127.1.87.67:39299)
0619 13:01:07.590220 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:07.590257 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:07.592869 (+ 2612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:07.593720 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:07.593731 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:07.594485 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:07.595407 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.595433 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.596935 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.596940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.599003 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.599014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.600159 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.600172 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.600405 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.601188 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:07.601230 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:07.604773 (+ 3543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:07.609098 (+ 4325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:07.609108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:07.609132 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:07.609592 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:07.610167 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:07.610174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:07.610179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:07.610470 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:07.611159 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:07.611172 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:07.611669 (+ 497us) client_negotiation.cc:769] Sending connection context
0619 13:01:07.612065 (+ 396us) client_negotiation.cc:240] Negotiation successful
0619 13:01:07.612627 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":454,"threads_started":1}
I20260619 13:01:07.646807 9322 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:37633
I20260619 13:01:07.646831 9456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:37633 every 8 connection(s)
I20260619 13:01:07.647766 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.630361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45629 (local address 127.1.87.67:37847)
0619 13:01:07.631600 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:07.631622 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:07.631776 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:07.632231 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:07.632239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:07.633074 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:07.633652 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.633671 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.635275 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.635281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.637212 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.637223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.638443 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.638447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.638606 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.639357 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:07.639390 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:07.640902 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:07.644945 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:07.644951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:07.644955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:07.645356 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:07.645752 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:07.645757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:07.645760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:07.645946 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:07.646623 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:07.646628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:07.646828 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 13:01:07.646974 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 13:01:07.647265 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":832,"threads_started":1}
I20260619 13:01:07.648075 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.630200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:37847 (local address 127.1.87.126:45629)
0619 13:01:07.630403 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:07.630410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:07.631703 (+ 1293us) server_negotiation.cc:407] Connection header received
0619 13:01:07.631925 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:07.631934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:07.632043 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:07.632161 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:07.633879 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.635107 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.637420 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.638055 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.638168 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.641126 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:07.641132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:07.641141 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:07.641173 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:07.644803 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.645489 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.645495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.645499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.645636 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.646112 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.646118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.646123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.646492 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:07.646660 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:07.647520 (+ 860us) server_negotiation.cc:299] Negotiation successful
0619 13:01:07.647759 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:01:07.648224 9322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:01:07.650170 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 9322
I20260619 13:01:07.650362 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:01:07.650574 9457 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45629
I20260619 13:01:07.650906 1373 external_mini_cluster.cc:1428] Setting key ed7c541057ee94abf8744a01e048ead0
I20260619 13:01:07.651078 9457 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:07.651918 9457 heartbeater.cc:511] Master 127.1.87.126:45629 requested a full tablet report, sending...
I20260619 13:01:07.653313 8971 ts_manager.cc:194] Registered new tserver with Master: 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:07.654170 8971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:39299
I20260619 13:01:07.666555 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:01:07 dist-test-slave-tpsm krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:01:07 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:07 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:01:07.715613 9067 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:07.705293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.1.87.126:45629)
0619 13:01:07.705518 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:07.705525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:07.705761 (+ 236us) server_negotiation.cc:407] Connection header received
0619 13:01:07.705976 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:07.705982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:07.706065 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:07.706171 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:07.707505 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.708229 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.709495 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.709918 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.710060 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.715384 (+ 5324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51024: BlockingRecv error: recv got EOF from 127.0.0.1:51024 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 13:01:07 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:01:07.731627 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.718332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.1.87.126:45629)
0619 13:01:07.718591 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:07.718600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:07.718885 (+ 285us) server_negotiation.cc:407] Connection header received
0619 13:01:07.719113 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:07.719117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:07.719179 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:07.719291 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:07.720540 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.721244 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.722540 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.722980 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.723160 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.726843 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:07.726847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:07.726851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:07.726880 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:07.729426 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.730080 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.730086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.730088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.730209 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:07.730517 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:07.730520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:07.730522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:07.730793 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:07.730990 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:07.731157 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 13:01:07.731388 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 13:01:07.735442 8971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:01:07.742199 8971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:01:07.776186 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.754077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.1.87.65:38991)
0619 13:01:07.766459 (+ 12382us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:07.766471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:07.766512 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:01:07.766644 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:07.766655 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:07.766996 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:07.768452 (+ 1456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:07.768687 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.770454 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.773389 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.774893 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.775017 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.775242 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:07.775696 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 13:01:07.775860 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12170,"thread_start_us":13142,"threads_started":1}
I20260619 13:01:07.776281 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.753912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:38991 (local address 127.0.0.1:47984)
0619 13:01:07.760975 (+ 7063us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:07.761013 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:07.761229 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:07.767562 (+ 6333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:07.767575 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:07.767630 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:07.768172 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.768189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.770607 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.770612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.773216 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.773231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.775101 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.775110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.775441 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.775507 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 13:01:07.775660 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 13:01:07.775856 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6796,"mutex_wait_us":56,"thread_start_us":6643,"threads_started":1}
I20260619 13:01:07.778008 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.761597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.1.87.66:38225)
0619 13:01:07.763923 (+ 2326us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:07.763935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:07.770725 (+ 6790us) server_negotiation.cc:407] Connection header received
0619 13:01:07.771114 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:07.771123 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:07.771356 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:07.771542 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:07.772162 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.773509 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.775517 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.776846 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.776966 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.777164 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:07.777574 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 13:01:07.777736 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2135,"thread_start_us":1789,"threads_started":1}
I20260619 13:01:07.779646 9477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.761568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:38225 (local address 127.0.0.1:40774)
0619 13:01:07.770611 (+ 9043us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:07.770629 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:07.770858 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:07.771551 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:07.771557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:07.771587 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:07.771989 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.772001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.773649 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.773657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.775365 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.775379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.777275 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.777279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.777391 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.777412 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 13:01:07.777518 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:01:07.777636 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8916,"mutex_wait_us":68,"thread_start_us":7438,"threads_started":1}
I20260619 13:01:07.780256 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.754123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:37633 (local address 127.0.0.1:53124)
0619 13:01:07.768739 (+ 14616us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:07.768765 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:07.768908 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:07.770607 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:07.770612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:07.770640 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:07.771401 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.771414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.773499 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.773504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.776956 (+ 3452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:07.776969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.779500 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.779507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:07.779674 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.779718 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 13:01:07.779800 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 13:01:07.779922 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14445,"mutex_wait_us":81,"thread_start_us":16224,"threads_started":1}
I20260619 13:01:07.780934 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:07.754524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.1.87.67:37633)
0619 13:01:07.770024 (+ 15500us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:07.770032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:07.770062 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:01:07.770163 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:07.770169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:07.770408 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:07.770608 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:07.771651 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.773369 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.777105 (+ 3736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:07.778368 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:07.778476 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:07.778619 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:07.779989 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0619 13:01:07.780607 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15355,"thread_start_us":15156,"threads_started":1}
I20260619 13:01:07.780402 9109 tablet_service.cc:1467] Processing CreateTablet for tablet fe767bf71ecc4ec7ae10481aad9a65c3 (DEFAULT_TABLE table=test-table [id=04f3bc93e6454f7d9df6d117fc4374dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:07.781885 9109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe767bf71ecc4ec7ae10481aad9a65c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:07.783103 9248 tablet_service.cc:1467] Processing CreateTablet for tablet fe767bf71ecc4ec7ae10481aad9a65c3 (DEFAULT_TABLE table=test-table [id=04f3bc93e6454f7d9df6d117fc4374dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:07.783700 9387 tablet_service.cc:1467] Processing CreateTablet for tablet fe767bf71ecc4ec7ae10481aad9a65c3 (DEFAULT_TABLE table=test-table [id=04f3bc93e6454f7d9df6d117fc4374dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:07.784386 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe767bf71ecc4ec7ae10481aad9a65c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:07.784778 9387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe767bf71ecc4ec7ae10481aad9a65c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:07.795548 9481 tablet_bootstrap.cc:492] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: Bootstrap starting.
I20260619 13:01:07.798542 9483 tablet_bootstrap.cc:492] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: Bootstrap starting.
I20260619 13:01:07.799811 9481 tablet_bootstrap.cc:654] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:07.801080 9481 log.cc:824] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:07.802031 9483 tablet_bootstrap.cc:654] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:07.802031 9482 tablet_bootstrap.cc:492] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: Bootstrap starting.
I20260619 13:01:07.803099 9483 log.cc:824] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:07.804569 9481 tablet_bootstrap.cc:492] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: No bootstrap required, opened a new log
I20260619 13:01:07.804822 9481 ts_tablet_manager.cc:1397] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 13:01:07.805158 9482 tablet_bootstrap.cc:654] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:07.806090 9483 tablet_bootstrap.cc:492] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: No bootstrap required, opened a new log
I20260619 13:01:07.806332 9483 ts_tablet_manager.cc:1397] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 13:01:07.806315 9482 log.cc:824] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:07.809460 9482 tablet_bootstrap.cc:492] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: No bootstrap required, opened a new log
I20260619 13:01:07.809688 9482 ts_tablet_manager.cc:1397] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 13:01:07.812382 9481 raft_consensus.cc:348] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:07.812734 9481 raft_consensus.cc:374] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:07.812831 9481 raft_consensus.cc:729] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261db81c10d545a99baeb865951226ac, State: Initialized, Role: FOLLOWER
I20260619 13:01:07.813382 9481 consensus_queue.cc:260] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:07.814414 9457 heartbeater.cc:503] Master 127.1.87.126:45629 was elected leader, sending a full tablet report...
I20260619 13:01:07.814920 9481 ts_tablet_manager.cc:1428] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260619 13:01:07.817245 9483 raft_consensus.cc:348] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:07.817667 9483 raft_consensus.cc:374] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:07.817771 9483 raft_consensus.cc:729] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf31521f80646bbadede2ce9289bb11, State: Initialized, Role: FOLLOWER
I20260619 13:01:07.818351 9483 consensus_queue.cc:260] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:07.819995 9318 heartbeater.cc:503] Master 127.1.87.126:45629 was elected leader, sending a full tablet report...
I20260619 13:01:07.820542 9483 ts_tablet_manager.cc:1428] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260619 13:01:07.819713 9482 raft_consensus.cc:348] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:07.821409 9482 raft_consensus.cc:374] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:07.821516 9482 raft_consensus.cc:729] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b28a41b03e41ee864e404b5c45f67e, State: Initialized, Role: FOLLOWER
I20260619 13:01:07.822094 9482 consensus_queue.cc:260] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:07.823817 9482 ts_tablet_manager.cc:1428] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260619 13:01:07.845043 9180 tablet.cc:2369] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:01:07.869760 9319 tablet.cc:2369] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:01:07.900202 9458 tablet.cc:2369] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:01:08.215575 9487 raft_consensus.cc:482] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:08.215787 9488 raft_consensus.cc:482] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:08.215927 9487 raft_consensus.cc:504] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.216180 9488 raft_consensus.cc:504] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.217654 9487 leader_election.cc:288] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225)
I20260619 13:01:08.222141 9488 leader_election.cc:288] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:08.238858 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.218150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:38991 (local address 127.1.87.66:55493)
0619 13:01:08.229635 (+ 11485us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.229663 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.229842 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.231497 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.231504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.231540 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:08.231913 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.231930 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.233764 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.233772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.236145 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.236154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.238053 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.238061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.238221 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.238268 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.238381 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.238510 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11283,"thread_start_us":15368,"threads_started":1}
I20260619 13:01:08.239576 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.218177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:55493 (local address 127.1.87.65:38991)
0619 13:01:08.230730 (+ 12553us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.230738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.230769 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:01:08.230857 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.230866 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.230960 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.231096 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:08.232150 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.233618 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.236327 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.237657 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.237788 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.237907 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.239095 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.239228 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":272,"server-negotiator.queue_time_us":12392,"thread_start_us":11904,"threads_started":1}
I20260619 13:01:08.240011 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.218163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:32873 (local address 127.1.87.65:38991)
0619 13:01:08.218421 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.218430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.230943 (+ 12513us) server_negotiation.cc:407] Connection header received
0619 13:01:08.231211 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.231216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.231352 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.231419 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:08.232464 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.233977 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.237225 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.238553 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.238656 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.238749 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.239635 (+ 886us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.239769 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 13:01:08.240770 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.217983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:38991 (local address 127.1.87.67:32873)
0619 13:01:08.230851 (+ 12868us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.230870 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.231043 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.231605 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.231611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.231644 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:08.232215 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.232234 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.234203 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.234211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.236929 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.236948 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.238904 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.238911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.239432 (+ 521us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.239474 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.240243 (+ 769us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.240378 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12763,"mutex_wait_us":244,"thread_start_us":12646,"threads_started":1}
I20260619 13:01:08.240556 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe767bf71ecc4ec7ae10481aad9a65c3" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e" is_pre_election: true
I20260619 13:01:08.240818 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.218463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:49321 (local address 127.1.87.67:37633)
0619 13:01:08.218683 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.218691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.233502 (+ 14811us) server_negotiation.cc:407] Connection header received
0619 13:01:08.233891 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.233897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.233972 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.234026 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:08.234853 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.235973 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.238290 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.239637 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.239762 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.239866 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.240395 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.240523 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":73}
I20260619 13:01:08.241173 9134 raft_consensus.cc:2454] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 0.
I20260619 13:01:08.241407 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.218877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:37633 (local address 127.1.87.66:49321)
0619 13:01:08.233413 (+ 14536us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.233429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.233739 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.234202 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.234207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.234230 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:08.234668 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.234679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.236145 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.236150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.238133 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.238144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.240020 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.240027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.240214 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.240247 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.240961 (+ 714us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.241079 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14431,"thread_start_us":11068,"threads_started":1}
I20260619 13:01:08.241379 9133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe767bf71ecc4ec7ae10481aad9a65c3" candidate_uuid: "261db81c10d545a99baeb865951226ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e" is_pre_election: true
I20260619 13:01:08.241703 9133 raft_consensus.cc:2454] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 261db81c10d545a99baeb865951226ac in term 0.
I20260619 13:01:08.242276 9202 leader_election.cc:302] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b28a41b03e41ee864e404b5c45f67e, aaf31521f80646bbadede2ce9289bb11; no voters:
I20260619 13:01:08.242697 9488 raft_consensus.cc:2781] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:01:08.242843 9488 raft_consensus.cc:482] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:01:08.242944 9488 raft_consensus.cc:3037] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:08.242355 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe767bf71ecc4ec7ae10481aad9a65c3" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac" is_pre_election: true
I20260619 13:01:08.243430 9341 leader_election.cc:302] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261db81c10d545a99baeb865951226ac, a0b28a41b03e41ee864e404b5c45f67e; no voters:
I20260619 13:01:08.243690 9412 raft_consensus.cc:2454] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 0.
I20260619 13:01:08.243907 9487 raft_consensus.cc:2781] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:01:08.244055 9487 raft_consensus.cc:482] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:01:08.244163 9487 raft_consensus.cc:3037] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:08.246222 9488 raft_consensus.cc:504] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.246321 9487 raft_consensus.cc:504] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.246547 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.231113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:57075 (local address 127.1.87.66:38225)
0619 13:01:08.231292 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.231297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.238076 (+ 6779us) server_negotiation.cc:407] Connection header received
0619 13:01:08.238505 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.238511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.238597 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.238661 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:08.239832 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.240879 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.244076 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.245332 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.245436 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.245533 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.246133 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.246279 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:01:08.246609 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.231459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:38225 (local address 127.1.87.67:57075)
0619 13:01:08.237986 (+ 6527us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.238003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.238356 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.238829 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.238833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.238858 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:08.239477 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.239488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.241728 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.241733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.243871 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.243885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.245727 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.245733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.245915 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.245963 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.246054 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.246190 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6401,"thread_start_us":6618,"threads_started":1}
I20260619 13:01:08.246935 9487 leader_election.cc:288] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [CANDIDATE]: Term 1 election: Requested vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225)
I20260619 13:01:08.246995 9488 leader_election.cc:288] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 election: Requested vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:08.247364 9133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe767bf71ecc4ec7ae10481aad9a65c3" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e"
I20260619 13:01:08.247453 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe767bf71ecc4ec7ae10481aad9a65c3" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac"
I20260619 13:01:08.247577 9133 raft_consensus.cc:3037] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:08.247700 9412 raft_consensus.cc:2379] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aaf31521f80646bbadede2ce9289bb11 in current term 1: Already voted for candidate 261db81c10d545a99baeb865951226ac in this term.
I20260619 13:01:08.249965 9133 raft_consensus.cc:2454] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 1.
I20260619 13:01:08.250356 9202 leader_election.cc:302] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0b28a41b03e41ee864e404b5c45f67e, aaf31521f80646bbadede2ce9289bb11; no voters: 261db81c10d545a99baeb865951226ac
I20260619 13:01:08.250660 9133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe767bf71ecc4ec7ae10481aad9a65c3" candidate_uuid: "261db81c10d545a99baeb865951226ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e"
I20260619 13:01:08.250941 9133 raft_consensus.cc:2379] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 261db81c10d545a99baeb865951226ac in current term 1: Already voted for candidate aaf31521f80646bbadede2ce9289bb11 in this term.
I20260619 13:01:08.251287 9267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe767bf71ecc4ec7ae10481aad9a65c3" candidate_uuid: "261db81c10d545a99baeb865951226ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf31521f80646bbadede2ce9289bb11"
I20260619 13:01:08.251500 9488 raft_consensus.cc:2781] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:08.251267 9268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe767bf71ecc4ec7ae10481aad9a65c3" candidate_uuid: "261db81c10d545a99baeb865951226ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf31521f80646bbadede2ce9289bb11" is_pre_election: true
I20260619 13:01:08.255586 9488 raft_consensus.cc:686] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 1 LEADER]: Becoming Leader. State: Replica: aaf31521f80646bbadede2ce9289bb11, State: Running, Role: LEADER
I20260619 13:01:08.256083 9488 consensus_queue.cc:237] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.259629 9341 leader_election.cc:302] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 261db81c10d545a99baeb865951226ac; no voters: a0b28a41b03e41ee864e404b5c45f67e, aaf31521f80646bbadede2ce9289bb11
I20260619 13:01:08.259940 9487 raft_consensus.cc:2726] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 13:01:08.260665 8971 catalog_manager.cc:5495] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf31521f80646bbadede2ce9289bb11 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "aaf31521f80646bbadede2ce9289bb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } health_report { overall_health: UNKNOWN } } }
I20260619 13:01:08.345283 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.338817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.1.87.66:38225)
0619 13:01:08.339068 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.339078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.339493 (+ 415us) server_negotiation.cc:407] Connection header received
0619 13:01:08.339677 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.339681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.339766 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.339895 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:01:08.340546 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.341297 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.342864 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.343253 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.343456 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.343863 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:01:08.344537 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:01:08.344653 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.344782 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.344848 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:01:08.352660 9133 raft_consensus.cc:1261] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Refusing update from remote peer aaf31521f80646bbadede2ce9289bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:08.352690 9412 raft_consensus.cc:1261] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Refusing update from remote peer aaf31521f80646bbadede2ce9289bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:08.353446 9495 consensus_queue.cc:1035] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:08.353763 9488 consensus_queue.cc:1035] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:08.369063 9502 mvcc.cc:205] Tried to move back new op lower bound from 7298556183963308032 to 7298556183580925952. Current Snapshot: MvccSnapshot[applied={T|T < 7298556183963308032 or (T in {7298556183963308032})}]
I20260619 13:01:08.380151 8972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:08.380366 8972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:08.383513 8972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78: Sending DeleteTablet for 3 replicas of tablet fe767bf71ecc4ec7ae10481aad9a65c3
I20260619 13:01:08.384752 9248 tablet_service.cc:1514] Processing DeleteTablet for tablet fe767bf71ecc4ec7ae10481aad9a65c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:08 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:40774
I20260619 13:01:08.384760 9387 tablet_service.cc:1514] Processing DeleteTablet for tablet fe767bf71ecc4ec7ae10481aad9a65c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:08 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:53124
I20260619 13:01:08.384874 9109 tablet_service.cc:1514] Processing DeleteTablet for tablet fe767bf71ecc4ec7ae10481aad9a65c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:08 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:47984
I20260619 13:01:08.386220 9513 tablet_replica.cc:330] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: stopping tablet replica
I20260619 13:01:08.386023 8972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 13:01:08.386628 9513 raft_consensus.cc:2229] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 1 LEADER]: Raft consensus shutting down.
W20260619 13:01:08.386700 8972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:01:08.387363 9513 raft_consensus.cc:2258] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:08.389290 9513 ts_tablet_manager.cc:1905] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:08.396494 9513 ts_tablet_manager.cc:1918] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:08.396677 9513 log.cc:1196] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/fe767bf71ecc4ec7ae10481aad9a65c3
I20260619 13:01:08.397027 9386 tablet_service.cc:1467] Processing CreateTablet for tablet 15c5d56ebad24a50850063c7c4e86a91 (DEFAULT_TABLE table=test-table [id=efc212102739487f8b231ecbd5f62f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:08.397205 9513 ts_tablet_manager.cc:1939] T fe767bf71ecc4ec7ae10481aad9a65c3 P aaf31521f80646bbadede2ce9289bb11: Deleting consensus metadata
I20260619 13:01:08.397600 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c5d56ebad24a50850063c7c4e86a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:08.399439 8959 catalog_manager.cc:4841] TS aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225): tablet fe767bf71ecc4ec7ae10481aad9a65c3 (table test-table [id=04f3bc93e6454f7d9df6d117fc4374dd]) successfully deleted
I20260619 13:01:08.400228 9108 tablet_service.cc:1467] Processing CreateTablet for tablet 15c5d56ebad24a50850063c7c4e86a91 (DEFAULT_TABLE table=test-table [id=efc212102739487f8b231ecbd5f62f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:08.400434 9248 tablet_service.cc:1467] Processing CreateTablet for tablet 15c5d56ebad24a50850063c7c4e86a91 (DEFAULT_TABLE table=test-table [id=efc212102739487f8b231ecbd5f62f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:08.400758 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c5d56ebad24a50850063c7c4e86a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:08.400825 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c5d56ebad24a50850063c7c4e86a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:08.403868 9514 tablet_replica.cc:330] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: stopping tablet replica
I20260619 13:01:08.404322 9514 raft_consensus.cc:2229] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:08.404878 9514 raft_consensus.cc:2258] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:08.405664 9515 tablet_replica.cc:330] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: stopping tablet replica
I20260619 13:01:08.406114 9515 raft_consensus.cc:2229] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:08.406611 9515 raft_consensus.cc:2258] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:08.413087 9518 tablet_bootstrap.cc:492] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: Bootstrap starting.
I20260619 13:01:08.414247 9514 ts_tablet_manager.cc:1905] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:08.415253 9518 tablet_bootstrap.cc:654] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:08.420086 9515 ts_tablet_manager.cc:1905] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:08.424265 9518 tablet_bootstrap.cc:492] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: No bootstrap required, opened a new log
I20260619 13:01:08.424546 9518 ts_tablet_manager.cc:1397] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260619 13:01:08.425627 9516 tablet_bootstrap.cc:492] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: Bootstrap starting.
I20260619 13:01:08.425592 9518 raft_consensus.cc:348] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.425976 9518 raft_consensus.cc:374] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:08.426080 9518 raft_consensus.cc:729] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b28a41b03e41ee864e404b5c45f67e, State: Initialized, Role: FOLLOWER
I20260619 13:01:08.426368 9518 consensus_queue.cc:260] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.427096 9518 ts_tablet_manager.cc:1428] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:01:08.429913 9516 tablet_bootstrap.cc:654] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:08.431928 9517 tablet_bootstrap.cc:492] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: Bootstrap starting.
I20260619 13:01:08.434214 9517 tablet_bootstrap.cc:654] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:08.435133 9514 ts_tablet_manager.cc:1918] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:08.435364 9514 log.cc:1196] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/fe767bf71ecc4ec7ae10481aad9a65c3
I20260619 13:01:08.435863 9514 ts_tablet_manager.cc:1939] T fe767bf71ecc4ec7ae10481aad9a65c3 P 261db81c10d545a99baeb865951226ac: Deleting consensus metadata
I20260619 13:01:08.436363 9515 ts_tablet_manager.cc:1918] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:08.436573 9515 log.cc:1196] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/fe767bf71ecc4ec7ae10481aad9a65c3
I20260619 13:01:08.437135 9515 ts_tablet_manager.cc:1939] T fe767bf71ecc4ec7ae10481aad9a65c3 P a0b28a41b03e41ee864e404b5c45f67e: Deleting consensus metadata
I20260619 13:01:08.437757 8956 catalog_manager.cc:4841] TS 261db81c10d545a99baeb865951226ac (127.1.87.67:37633): tablet fe767bf71ecc4ec7ae10481aad9a65c3 (table test-table [id=04f3bc93e6454f7d9df6d117fc4374dd]) successfully deleted
I20260619 13:01:08.438864 8959 catalog_manager.cc:4841] TS a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991): tablet fe767bf71ecc4ec7ae10481aad9a65c3 (table test-table [id=04f3bc93e6454f7d9df6d117fc4374dd]) successfully deleted
I20260619 13:01:08.440982 9517 tablet_bootstrap.cc:492] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: No bootstrap required, opened a new log
I20260619 13:01:08.441186 9517 ts_tablet_manager.cc:1397] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260619 13:01:08.442155 9517 raft_consensus.cc:348] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.442369 9517 raft_consensus.cc:374] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:08.442543 9517 raft_consensus.cc:729] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf31521f80646bbadede2ce9289bb11, State: Initialized, Role: FOLLOWER
I20260619 13:01:08.442814 9517 consensus_queue.cc:260] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.443480 9517 ts_tablet_manager.cc:1428] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 13:01:08.444388 9516 tablet_bootstrap.cc:492] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: No bootstrap required, opened a new log
I20260619 13:01:08.444545 9516 ts_tablet_manager.cc:1397] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
I20260619 13:01:08.445303 9516 raft_consensus.cc:348] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.445549 9516 raft_consensus.cc:374] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:08.445652 9516 raft_consensus.cc:729] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261db81c10d545a99baeb865951226ac, State: Initialized, Role: FOLLOWER
I20260619 13:01:08.445922 9516 consensus_queue.cc:260] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.447141 9516 ts_tablet_manager.cc:1428] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 13:01:08.715190 9503 raft_consensus.cc:482] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:08.715551 9503 raft_consensus.cc:504] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.717597 9495 raft_consensus.cc:482] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:08.717826 9495 raft_consensus.cc:504] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.718035 9503 leader_election.cc:288] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:08.718523 9495 leader_election.cc:288] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:08.719213 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c5d56ebad24a50850063c7c4e86a91" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac" is_pre_election: true
I20260619 13:01:08.720400 9412 raft_consensus.cc:2454] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 0.
I20260619 13:01:08.720918 9199 leader_election.cc:302] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261db81c10d545a99baeb865951226ac, aaf31521f80646bbadede2ce9289bb11; no voters:
I20260619 13:01:08.721261 9495 raft_consensus.cc:2781] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:01:08.721374 9495 raft_consensus.cc:482] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:01:08.721490 9495 raft_consensus.cc:3037] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:08.724720 9495 raft_consensus.cc:504] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.726119 9133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c5d56ebad24a50850063c7c4e86a91" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e" is_pre_election: true
I20260619 13:01:08.726401 9133 raft_consensus.cc:2454] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 0.
I20260619 13:01:08.726755 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c5d56ebad24a50850063c7c4e86a91" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e"
I20260619 13:01:08.726925 9134 raft_consensus.cc:3037] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:08.727291 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.718398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:53575 (local address 127.1.87.66:38225)
0619 13:01:08.718716 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.718724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.719641 (+ 917us) server_negotiation.cc:407] Connection header received
0619 13:01:08.719952 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.719960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.720055 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.720119 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:08.720888 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.722064 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.724705 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.726023 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.726127 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.726230 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.726849 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.727007 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260619 13:01:08.728019 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.718415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:38225 (local address 127.1.87.65:53575)
0619 13:01:08.719546 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.719570 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.719763 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.720314 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.720319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.720343 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:08.720730 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.720742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.722237 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.722242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.724496 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.724513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.726459 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.726465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.726641 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.726685 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.727509 (+ 824us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.727655 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":800,"threads_started":1}
I20260619 13:01:08.728417 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.719980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:51463 (local address 127.1.87.67:37633)
0619 13:01:08.720150 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.720156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.720305 (+ 149us) server_negotiation.cc:407] Connection header received
0619 13:01:08.720701 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.720707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.720789 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.720853 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:08.721834 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.722920 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.725647 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.726905 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.727005 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.727096 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.728034 (+ 938us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.728167 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:01:08.729204 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.719647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:37633 (local address 127.1.87.65:51463)
0619 13:01:08.720204 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.720224 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.720470 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.721044 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.721119 (+ 75us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.721146 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:08.721627 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.721643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.723101 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.723107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.725446 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.725460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.727618 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.727625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.727829 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.727865 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.728719 (+ 854us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.728864 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":5784,"threads_started":1}
I20260619 13:01:08.730067 9134 raft_consensus.cc:2454] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 1.
I20260619 13:01:08.730873 9267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c5d56ebad24a50850063c7c4e86a91" candidate_uuid: "a0b28a41b03e41ee864e404b5c45f67e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf31521f80646bbadede2ce9289bb11" is_pre_election: true
I20260619 13:01:08.731204 9202 leader_election.cc:302] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b28a41b03e41ee864e404b5c45f67e, aaf31521f80646bbadede2ce9289bb11; no voters:
I20260619 13:01:08.731272 9267 raft_consensus.cc:2379] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0b28a41b03e41ee864e404b5c45f67e in current term 1: Already voted for candidate aaf31521f80646bbadede2ce9289bb11 in this term.
I20260619 13:01:08.731441 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c5d56ebad24a50850063c7c4e86a91" candidate_uuid: "a0b28a41b03e41ee864e404b5c45f67e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac" is_pre_election: true
I20260619 13:01:08.731700 9412 raft_consensus.cc:2454] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0b28a41b03e41ee864e404b5c45f67e in term 0.
I20260619 13:01:08.732014 9508 raft_consensus.cc:2781] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:08.732234 9508 raft_consensus.cc:686] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 1 LEADER]: Becoming Leader. State: Replica: aaf31521f80646bbadede2ce9289bb11, State: Running, Role: LEADER
I20260619 13:01:08.732160 9059 leader_election.cc:302] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 261db81c10d545a99baeb865951226ac, a0b28a41b03e41ee864e404b5c45f67e; no voters: aaf31521f80646bbadede2ce9289bb11
I20260619 13:01:08.732611 9503 raft_consensus.cc:2741] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 13:01:08.732586 9508 consensus_queue.cc:237] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.734455 9495 leader_election.cc:288] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 election: Requested vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:08.734910 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c5d56ebad24a50850063c7c4e86a91" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac"
I20260619 13:01:08.735138 9412 raft_consensus.cc:3037] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:08.736020 8971 catalog_manager.cc:5495] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf31521f80646bbadede2ce9289bb11 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "aaf31521f80646bbadede2ce9289bb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } health_report { overall_health: UNKNOWN } } }
I20260619 13:01:08.737260 9412 raft_consensus.cc:2454] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 1.
Jun 19 13:01:08 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874065, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.126@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:01:08.825214 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.809608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45629 (local address 127.0.0.1:51052)
0619 13:01:08.810299 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.810318 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.810467 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.810892 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.810897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.811740 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:08.812044 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.812057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.813220 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.813226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.814839 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.814857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.815742 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.815750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.815955 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.816732 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:08.816756 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:08.819538 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:08.822460 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:08.822473 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:08.822481 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:08.822883 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:08.823246 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:08.823251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:08.823255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:08.823510 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:08.824072 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:08.824080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:08.824253 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.824414 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.824720 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":359,"threads_started":1}
I20260619 13:01:08.825237 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.809824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.1.87.126:45629)
0619 13:01:08.810610 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.810614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.810633 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:01:08.810675 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.810678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.810731 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.810817 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:08.812224 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.812991 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.814992 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.815563 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.815663 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.819743 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:08.819754 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:08.819757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:08.819784 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:08.822335 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:08.822997 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:08.823006 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:08.823009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:08.823142 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:08.823626 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:08.823632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:08.823636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:08.823933 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:08.824084 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.824496 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.824840 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":694,"thread_start_us":560,"threads_started":1}
I20260619 13:01:08.830480 8971 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:51052:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 13:01:08.831554 8971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:01:08.838932 9108 tablet_service.cc:1467] Processing CreateTablet for tablet f30d9aa2df204d149cd3e24d2e6708e3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f996be4a05743c595cfc37038e85924]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:01:08.838973 9386 tablet_service.cc:1467] Processing CreateTablet for tablet f30d9aa2df204d149cd3e24d2e6708e3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f996be4a05743c595cfc37038e85924]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:01:08.838977 9248 tablet_service.cc:1467] Processing CreateTablet for tablet f30d9aa2df204d149cd3e24d2e6708e3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f996be4a05743c595cfc37038e85924]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:01:08.839509 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30d9aa2df204d149cd3e24d2e6708e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:08.839622 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30d9aa2df204d149cd3e24d2e6708e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:08.839634 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30d9aa2df204d149cd3e24d2e6708e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:08.846611 9516 tablet_bootstrap.cc:492] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: Bootstrap starting.
I20260619 13:01:08.846622 9517 tablet_bootstrap.cc:492] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: Bootstrap starting.
I20260619 13:01:08.847740 9518 tablet_bootstrap.cc:492] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: Bootstrap starting.
I20260619 13:01:08.848348 9517 tablet_bootstrap.cc:654] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:08.848584 9516 tablet_bootstrap.cc:654] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:08.849457 9518 tablet_bootstrap.cc:654] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:08.850984 9517 tablet_bootstrap.cc:492] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: No bootstrap required, opened a new log
I20260619 13:01:08.851156 9517 ts_tablet_manager.cc:1397] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 13:01:08.851464 9518 tablet_bootstrap.cc:492] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: No bootstrap required, opened a new log
I20260619 13:01:08.851621 9518 ts_tablet_manager.cc:1397] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:01:08.851995 9517 raft_consensus.cc:348] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.852243 9516 tablet_bootstrap.cc:492] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: No bootstrap required, opened a new log
I20260619 13:01:08.852265 9517 raft_consensus.cc:374] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:08.852346 9517 raft_consensus.cc:729] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf31521f80646bbadede2ce9289bb11, State: Initialized, Role: FOLLOWER
I20260619 13:01:08.852392 9516 ts_tablet_manager.cc:1397] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260619 13:01:08.852279 9518 raft_consensus.cc:348] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.852550 9518 raft_consensus.cc:374] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:08.852638 9518 raft_consensus.cc:729] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b28a41b03e41ee864e404b5c45f67e, State: Initialized, Role: FOLLOWER
I20260619 13:01:08.852586 9517 consensus_queue.cc:260] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.852855 9518 consensus_queue.cc:260] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.853096 9517 ts_tablet_manager.cc:1428] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:01:08.853089 9516 raft_consensus.cc:348] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.853331 9516 raft_consensus.cc:374] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:08.853404 9516 raft_consensus.cc:729] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261db81c10d545a99baeb865951226ac, State: Initialized, Role: FOLLOWER
I20260619 13:01:08.853271 9503 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.853544 9503 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:01:08.853600 9495 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.853613 9516 consensus_queue.cc:260] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.853834 9495 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:01:08.854118 9516 ts_tablet_manager.cc:1428] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 13:01:08.854341 9518 ts_tablet_manager.cc:1428] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 13:01:08.854399 9487 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.854676 9487 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:01:08.858112 9503 raft_consensus.cc:482] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:08.858258 9503 raft_consensus.cc:504] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.858781 9503 leader_election.cc:288] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:08.859103 9267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30d9aa2df204d149cd3e24d2e6708e3" candidate_uuid: "a0b28a41b03e41ee864e404b5c45f67e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf31521f80646bbadede2ce9289bb11" is_pre_election: true
I20260619 13:01:08.859159 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30d9aa2df204d149cd3e24d2e6708e3" candidate_uuid: "a0b28a41b03e41ee864e404b5c45f67e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac" is_pre_election: true
I20260619 13:01:08.859367 9412 raft_consensus.cc:2454] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0b28a41b03e41ee864e404b5c45f67e in term 0.
I20260619 13:01:08.859387 9267 raft_consensus.cc:2454] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0b28a41b03e41ee864e404b5c45f67e in term 0.
I20260619 13:01:08.859740 9059 leader_election.cc:302] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261db81c10d545a99baeb865951226ac, a0b28a41b03e41ee864e404b5c45f67e; no voters:
I20260619 13:01:08.860005 9503 raft_consensus.cc:2781] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:01:08.860136 9503 raft_consensus.cc:482] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:01:08.860260 9503 raft_consensus.cc:3037] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:08.862576 9503 raft_consensus.cc:504] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.863082 9503 leader_election.cc:288] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 election: Requested vote from peers aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:08.863598 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30d9aa2df204d149cd3e24d2e6708e3" candidate_uuid: "a0b28a41b03e41ee864e404b5c45f67e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac"
I20260619 13:01:08.863786 9412 raft_consensus.cc:3037] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:08.866240 9412 raft_consensus.cc:2454] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0b28a41b03e41ee864e404b5c45f67e in term 1.
I20260619 13:01:08.866582 9267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30d9aa2df204d149cd3e24d2e6708e3" candidate_uuid: "a0b28a41b03e41ee864e404b5c45f67e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf31521f80646bbadede2ce9289bb11"
I20260619 13:01:08.866683 9059 leader_election.cc:302] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261db81c10d545a99baeb865951226ac, a0b28a41b03e41ee864e404b5c45f67e; no voters:
I20260619 13:01:08.866791 9267 raft_consensus.cc:3037] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:08.866981 9503 raft_consensus.cc:2781] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:08.867483 9503 raft_consensus.cc:686] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [term 1 LEADER]: Becoming Leader. State: Replica: a0b28a41b03e41ee864e404b5c45f67e, State: Running, Role: LEADER
I20260619 13:01:08.867908 9503 consensus_queue.cc:237] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:08.869088 9267 raft_consensus.cc:2454] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0b28a41b03e41ee864e404b5c45f67e in term 1.
I20260619 13:01:08.870709 9535 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: TxnStatusTablet state changed. Reason: New leader a0b28a41b03e41ee864e404b5c45f67e. Latest consensus state: current_term: 1 leader_uuid: "a0b28a41b03e41ee864e404b5c45f67e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.870992 9535 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:01:08.871665 9537 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 13:01:08.871822 9537 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 13:01:08.871626 8971 catalog_manager.cc:5495] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e reported cstate change: term changed from 0 to 1, leader changed from <none> to a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65). New cstate: current_term: 1 leader_uuid: "a0b28a41b03e41ee864e404b5c45f67e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } health_report { overall_health: UNKNOWN } } }
I20260619 13:01:08.894392 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.885596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:38991 (local address 127.0.0.1:47998)
0619 13:01:08.885857 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.885879 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.886023 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.888129 (+ 2106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.888134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.888172 (+ 38us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:01:08.888555 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.888571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.889891 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.889898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.891576 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.891592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.892456 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.892463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.892688 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.892766 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:01:08.893788 (+ 1022us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:01:08.893810 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.893909 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.894067 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 13:01:08.894338 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.885609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.1.87.65:38991)
0619 13:01:08.887773 (+ 2164us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.887777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.887793 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:01:08.887865 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.887869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.887940 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.888071 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:01:08.888777 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.889760 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.891767 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.892221 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.892441 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.893022 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:01:08.893664 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:01:08.893787 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.893957 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.894034 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2042,"thread_start_us":1893,"threads_started":1}
I20260619 13:01:08.899245 9267 raft_consensus.cc:1261] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Refusing update from remote peer a0b28a41b03e41ee864e404b5c45f67e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:08.899362 9412 raft_consensus.cc:1261] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Refusing update from remote peer a0b28a41b03e41ee864e404b5c45f67e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:08.900007 9503 consensus_queue.cc:1035] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:08.900293 9535 consensus_queue.cc:1035] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e [LEADER]: Connected to new peer: Peer: permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:08.908092 9487 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: TxnStatusTablet state changed. Reason: New leader a0b28a41b03e41ee864e404b5c45f67e. Latest consensus state: current_term: 1 leader_uuid: "a0b28a41b03e41ee864e404b5c45f67e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.908373 9487 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:01:08.909648 9495 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: TxnStatusTablet state changed. Reason: New leader a0b28a41b03e41ee864e404b5c45f67e. Latest consensus state: current_term: 1 leader_uuid: "a0b28a41b03e41ee864e404b5c45f67e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.909950 9495 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:01:08.910933 9541 mvcc.cc:205] Tried to move back new op lower bound from 7298556186203041792 to 7298556186086686720. Current Snapshot: MvccSnapshot[applied={T|T < 7298556186203041792}]
I20260619 13:01:08.911370 9542 mvcc.cc:205] Tried to move back new op lower bound from 7298556186203041792 to 7298556186086686720. Current Snapshot: MvccSnapshot[applied={T|T < 7298556186203041792}]
I20260619 13:01:08.922269 9535 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0b28a41b03e41ee864e404b5c45f67e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.922640 9535 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:01:08.926121 9535 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0b28a41b03e41ee864e404b5c45f67e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.926487 9535 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P a0b28a41b03e41ee864e404b5c45f67e: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:01:08.930291 9487 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0b28a41b03e41ee864e404b5c45f67e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.930537 9487 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P 261db81c10d545a99baeb865951226ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:01:08.934278 9495 tablet_replica.cc:439] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0b28a41b03e41ee864e404b5c45f67e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } } }
I20260619 13:01:08.934552 9495 tablet_replica.cc:442] T f30d9aa2df204d149cd3e24d2e6708e3 P aaf31521f80646bbadede2ce9289bb11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:01:08.950551 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.944664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:38991 (local address 127.1.87.66:43817)
0619 13:01:08.945281 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.945297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.945411 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.945752 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.945755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.945764 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:01:08.946006 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.946013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.947406 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.947409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.948573 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.948584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.949532 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.949536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.949665 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.949724 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:01:08.950118 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:01:08.950125 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.950195 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.950286 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":312,"threads_started":1}
I20260619 13:01:08.950599 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.944731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:43817 (local address 127.1.87.65:38991)
0619 13:01:08.944925 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.944932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.945377 (+ 445us) server_negotiation.cc:407] Connection header received
0619 13:01:08.945542 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.945548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.945634 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.945725 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:01:08.946150 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.947268 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.948711 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.949370 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.949507 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.949887 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:01:08.950024 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:01:08.950112 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.950230 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.950311 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:01:08.958741 9412 raft_consensus.cc:1261] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Refusing update from remote peer aaf31521f80646bbadede2ce9289bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:08.958950 9134 raft_consensus.cc:1261] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Refusing update from remote peer aaf31521f80646bbadede2ce9289bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:08.959281 9508 consensus_queue.cc:1035] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:08.959550 9495 consensus_queue.cc:1035] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:08.998179 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.989105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:42431 (local address 127.1.87.66:38225)
0619 13:01:08.989324 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:08.989332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:08.989919 (+ 587us) server_negotiation.cc:407] Connection header received
0619 13:01:08.990221 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:08.990229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:08.990307 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:08.990366 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:01:08.991186 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.992318 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.995636 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.996302 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.996394 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.997051 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:01:08.997180 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:01:08.997238 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:08.997805 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 13:01:08.997885 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:01:08.998940 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:08.989050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:38225 (local address 127.1.87.65:42431)
0619 13:01:08.989819 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:08.989843 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:08.990066 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:08.990559 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:08.990564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:08.990591 (+ 27us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:01:08.991014 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.991027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.992480 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.992486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.995467 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:08.995481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:08.996596 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:08.996603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:08.996784 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:08.996854 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:01:08.997325 (+ 471us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:01:08.997340 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 13:01:08.998405 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0619 13:01:08.998572 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":2264,"threads_started":1}
I20260619 13:01:09.040328 8971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:09.040587 8971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:09.043498 8971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78: Sending DeleteTablet for 3 replicas of tablet 15c5d56ebad24a50850063c7c4e86a91
I20260619 13:01:09.044231 9108 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c5d56ebad24a50850063c7c4e86a91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:47984
I20260619 13:01:09.044320 9248 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c5d56ebad24a50850063c7c4e86a91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:40774
I20260619 13:01:09.044365 9386 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c5d56ebad24a50850063c7c4e86a91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:53124
I20260619 13:01:09.060025 9558 tablet_replica.cc:330] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: stopping tablet replica
I20260619 13:01:09.060303 9558 raft_consensus.cc:2229] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:09.060628 9558 raft_consensus.cc:2258] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:09.063519 9558 ts_tablet_manager.cc:1905] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:09.067724 9560 tablet_replica.cc:330] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: stopping tablet replica
I20260619 13:01:09.067919 9560 raft_consensus.cc:2229] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:09.068210 9560 raft_consensus.cc:2258] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:09.070174 9560 ts_tablet_manager.cc:1905] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:09.072154 9558 ts_tablet_manager.cc:1918] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:01:09.072181 9559 tablet_replica.cc:330] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: stopping tablet replica
I20260619 13:01:09.072314 9558 log.cc:1196] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/15c5d56ebad24a50850063c7c4e86a91
I20260619 13:01:09.072592 9559 raft_consensus.cc:2229] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:01:09.072798 9558 ts_tablet_manager.cc:1939] T 15c5d56ebad24a50850063c7c4e86a91 P a0b28a41b03e41ee864e404b5c45f67e: Deleting consensus metadata
I20260619 13:01:09.073104 9559 raft_consensus.cc:2258] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:09.073946 8959 catalog_manager.cc:4841] TS a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991): tablet 15c5d56ebad24a50850063c7c4e86a91 (table test-table [id=efc212102739487f8b231ecbd5f62f2d]) successfully deleted
I20260619 13:01:09.074239 9559 ts_tablet_manager.cc:1905] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:09.077526 9560 ts_tablet_manager.cc:1918] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:01:09.077653 9560 log.cc:1196] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/15c5d56ebad24a50850063c7c4e86a91
I20260619 13:01:09.078088 9560 ts_tablet_manager.cc:1939] T 15c5d56ebad24a50850063c7c4e86a91 P 261db81c10d545a99baeb865951226ac: Deleting consensus metadata
I20260619 13:01:09.079200 8956 catalog_manager.cc:4841] TS 261db81c10d545a99baeb865951226ac (127.1.87.67:37633): tablet 15c5d56ebad24a50850063c7c4e86a91 (table test-table [id=efc212102739487f8b231ecbd5f62f2d]) successfully deleted
I20260619 13:01:09.081629 9559 ts_tablet_manager.cc:1918] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:01:09.081755 9559 log.cc:1196] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/15c5d56ebad24a50850063c7c4e86a91
I20260619 13:01:09.082195 9559 ts_tablet_manager.cc:1939] T 15c5d56ebad24a50850063c7c4e86a91 P aaf31521f80646bbadede2ce9289bb11: Deleting consensus metadata
I20260619 13:01:09.083204 8959 catalog_manager.cc:4841] TS aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225): tablet 15c5d56ebad24a50850063c7c4e86a91 (table test-table [id=efc212102739487f8b231ecbd5f62f2d]) successfully deleted
Jun 19 13:01:09 dist-test-slave-tpsm krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:01:09 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:09 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:01:09.119915 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:09.107143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.1.87.126:45629)
0619 13:01:09.107420 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:09.107430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:09.107821 (+ 391us) server_negotiation.cc:407] Connection header received
0619 13:01:09.107990 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:09.107994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:09.108058 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:09.108143 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:09.109985 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.111145 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.112979 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.113651 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.113780 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:09.119667 (+ 5887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51054: BlockingRecv error: recv got EOF from 127.0.0.1:51054 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 13:01:09 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM
I20260619 13:01:09.141605 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:09.124657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.1.87.126:45629)
0619 13:01:09.124831 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:09.124837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:09.124891 (+ 54us) server_negotiation.cc:407] Connection header received
0619 13:01:09.125125 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:09.125131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:09.125213 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:09.125308 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:09.126720 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.127924 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.129742 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.130439 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.130556 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:09.135169 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:09.135176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:09.135181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:09.135249 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:09.138796 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:09.139592 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:09.139597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:09.139601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:09.139743 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:09.140125 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:09.140130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:09.140132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:09.140471 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:09.140736 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:09.141024 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 13:01:09.141246 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:01:09.145390 8971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:01:09.146092 8971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:01:09.152673 9108 tablet_service.cc:1467] Processing CreateTablet for tablet 08df1bda83254e71bb642c7eb18b58ab (DEFAULT_TABLE table=test-table [id=3e89dc92f225469b94860befa2989112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:09.153153 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08df1bda83254e71bb642c7eb18b58ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:09.159737 9248 tablet_service.cc:1467] Processing CreateTablet for tablet 08df1bda83254e71bb642c7eb18b58ab (DEFAULT_TABLE table=test-table [id=3e89dc92f225469b94860befa2989112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:09.160171 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08df1bda83254e71bb642c7eb18b58ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:09.163719 9386 tablet_service.cc:1467] Processing CreateTablet for tablet 08df1bda83254e71bb642c7eb18b58ab (DEFAULT_TABLE table=test-table [id=3e89dc92f225469b94860befa2989112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:09.164132 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08df1bda83254e71bb642c7eb18b58ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:09.166549 9517 tablet_bootstrap.cc:492] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: Bootstrap starting.
I20260619 13:01:09.168107 9517 tablet_bootstrap.cc:654] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:09.168897 9516 tablet_bootstrap.cc:492] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: Bootstrap starting.
I20260619 13:01:09.169474 9518 tablet_bootstrap.cc:492] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: Bootstrap starting.
I20260619 13:01:09.170488 9516 tablet_bootstrap.cc:654] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:09.171233 9518 tablet_bootstrap.cc:654] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:09.180074 9517 tablet_bootstrap.cc:492] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: No bootstrap required, opened a new log
I20260619 13:01:09.180248 9517 ts_tablet_manager.cc:1397] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260619 13:01:09.181078 9517 raft_consensus.cc:348] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.181345 9517 raft_consensus.cc:374] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:09.181442 9517 raft_consensus.cc:729] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf31521f80646bbadede2ce9289bb11, State: Initialized, Role: FOLLOWER
I20260619 13:01:09.181721 9517 consensus_queue.cc:260] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.182375 9516 tablet_bootstrap.cc:492] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: No bootstrap required, opened a new log
I20260619 13:01:09.182535 9516 ts_tablet_manager.cc:1397] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20260619 13:01:09.182952 9517 ts_tablet_manager.cc:1428] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 13:01:09.183331 9516 raft_consensus.cc:348] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.183580 9516 raft_consensus.cc:374] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:09.183696 9516 raft_consensus.cc:729] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261db81c10d545a99baeb865951226ac, State: Initialized, Role: FOLLOWER
I20260619 13:01:09.183915 9516 consensus_queue.cc:260] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.184463 9516 ts_tablet_manager.cc:1428] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 13:01:09.184960 9518 tablet_bootstrap.cc:492] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: No bootstrap required, opened a new log
I20260619 13:01:09.185117 9518 ts_tablet_manager.cc:1397] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260619 13:01:09.185935 9518 raft_consensus.cc:348] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.186242 9518 raft_consensus.cc:374] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:09.186367 9518 raft_consensus.cc:729] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b28a41b03e41ee864e404b5c45f67e, State: Initialized, Role: FOLLOWER
I20260619 13:01:09.186604 9518 consensus_queue.cc:260] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.187268 9518 ts_tablet_manager.cc:1428] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:01:09.196013 9488 raft_consensus.cc:482] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:09.196211 9488 raft_consensus.cc:504] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.196848 9488 leader_election.cc:288] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:09.197314 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df1bda83254e71bb642c7eb18b58ab" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e" is_pre_election: true
I20260619 13:01:09.197364 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df1bda83254e71bb642c7eb18b58ab" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac" is_pre_election: true
I20260619 13:01:09.197543 9134 raft_consensus.cc:2454] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 0.
I20260619 13:01:09.197672 9412 raft_consensus.cc:2454] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 0.
I20260619 13:01:09.197925 9202 leader_election.cc:302] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b28a41b03e41ee864e404b5c45f67e, aaf31521f80646bbadede2ce9289bb11; no voters:
I20260619 13:01:09.198252 9488 raft_consensus.cc:2781] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:01:09.198374 9488 raft_consensus.cc:482] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:01:09.198486 9488 raft_consensus.cc:3037] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:09.200503 9488 raft_consensus.cc:504] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.201046 9488 leader_election.cc:288] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 election: Requested vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:09.201496 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df1bda83254e71bb642c7eb18b58ab" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e"
I20260619 13:01:09.201486 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08df1bda83254e71bb642c7eb18b58ab" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac"
I20260619 13:01:09.201701 9134 raft_consensus.cc:3037] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:09.201706 9412 raft_consensus.cc:3037] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:09.203876 9134 raft_consensus.cc:2454] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 1.
I20260619 13:01:09.204280 9202 leader_election.cc:302] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b28a41b03e41ee864e404b5c45f67e, aaf31521f80646bbadede2ce9289bb11; no voters:
I20260619 13:01:09.204541 9488 raft_consensus.cc:2781] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:09.204655 9412 raft_consensus.cc:2454] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 1.
I20260619 13:01:09.204713 9488 raft_consensus.cc:686] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 1 LEADER]: Becoming Leader. State: Replica: aaf31521f80646bbadede2ce9289bb11, State: Running, Role: LEADER
I20260619 13:01:09.204993 9488 consensus_queue.cc:237] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.207746 8971 catalog_manager.cc:5495] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf31521f80646bbadede2ce9289bb11 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "aaf31521f80646bbadede2ce9289bb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } health_report { overall_health: UNKNOWN } } }
I20260619 13:01:09.230078 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:09.224835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.1.87.66:38225)
0619 13:01:09.225014 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:09.225020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:09.225158 (+ 138us) server_negotiation.cc:407] Connection header received
0619 13:01:09.225374 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:09.225379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:09.225457 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:09.225552 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:01:09.225918 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.226700 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.228074 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.228578 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.228701 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:09.229228 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:01:09.229340 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:01:09.229408 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:09.229600 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 13:01:09.229678 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:01:09.232815 9134 raft_consensus.cc:1261] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Refusing update from remote peer aaf31521f80646bbadede2ce9289bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:09.233444 9412 raft_consensus.cc:1261] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Refusing update from remote peer aaf31521f80646bbadede2ce9289bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:09.233451 9495 consensus_queue.cc:1035] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:09.233956 9488 consensus_queue.cc:1035] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:09.246574 8971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:09.246886 8971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:09.251106 8971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78: Sending DeleteTablet for 3 replicas of tablet 08df1bda83254e71bb642c7eb18b58ab
I20260619 13:01:09.252310 9386 tablet_service.cc:1514] Processing DeleteTablet for tablet 08df1bda83254e71bb642c7eb18b58ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:53124
I20260619 13:01:09.252619 9560 tablet_replica.cc:330] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: stopping tablet replica
I20260619 13:01:09.252825 9560 raft_consensus.cc:2229] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:09.252904 9108 tablet_service.cc:1514] Processing DeleteTablet for tablet 08df1bda83254e71bb642c7eb18b58ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:47984
I20260619 13:01:09.253055 9560 raft_consensus.cc:2258] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:09.253327 9109 tablet_service.cc:1514] Processing DeleteTablet for tablet 08df1bda83254e71bb642c7eb18b58ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:47984
I20260619 13:01:09.253597 9248 tablet_service.cc:1514] Processing DeleteTablet for tablet 08df1bda83254e71bb642c7eb18b58ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:40774
I20260619 13:01:09.253647 9247 tablet_service.cc:1514] Processing DeleteTablet for tablet 08df1bda83254e71bb642c7eb18b58ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:40774
I20260619 13:01:09.253407 9558 tablet_replica.cc:330] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: stopping tablet replica
I20260619 13:01:09.253892 9559 tablet_replica.cc:330] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: stopping tablet replica
I20260619 13:01:09.254068 9559 raft_consensus.cc:2229] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:01:09.254079 9558 raft_consensus.cc:2229] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:09.254267 9558 raft_consensus.cc:2258] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:09.254345 9559 raft_consensus.cc:2258] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:09.255250 9559 ts_tablet_manager.cc:1905] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:09.255445 8972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 13:01:09.255856 9558 ts_tablet_manager.cc:1905] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 13:01:09.256270 8972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:01:09.261900 9559 ts_tablet_manager.cc:1918] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:09.262042 9559 log.cc:1196] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/08df1bda83254e71bb642c7eb18b58ab
I20260619 13:01:09.262458 9559 ts_tablet_manager.cc:1939] T 08df1bda83254e71bb642c7eb18b58ab P aaf31521f80646bbadede2ce9289bb11: Deleting consensus metadata
I20260619 13:01:09.262924 9558 ts_tablet_manager.cc:1918] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:09.263139 9558 log.cc:1196] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/08df1bda83254e71bb642c7eb18b58ab
I20260619 13:01:09.263659 9558 ts_tablet_manager.cc:1939] T 08df1bda83254e71bb642c7eb18b58ab P a0b28a41b03e41ee864e404b5c45f67e: Deleting consensus metadata
I20260619 13:01:09.263934 9386 tablet_service.cc:1467] Processing CreateTablet for tablet 1444986ee92e4c70b16cb30d0d906157 (DEFAULT_TABLE table=test-table [id=14cab0de4c1d442b946cc3eb9acf30a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:09.264333 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1444986ee92e4c70b16cb30d0d906157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:09.265551 9560 ts_tablet_manager.cc:1905] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:09.267551 8959 catalog_manager.cc:4841] TS a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991): tablet 08df1bda83254e71bb642c7eb18b58ab (table test-table [id=3e89dc92f225469b94860befa2989112]) successfully deleted
W20260619 13:01:09.267964 8959 catalog_manager.cc:4808] TS a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991): delete failed for tablet 08df1bda83254e71bb642c7eb18b58ab because the tablet was not found. No further retry: Not found: Tablet not found: 08df1bda83254e71bb642c7eb18b58ab
I20260619 13:01:09.268083 9109 tablet_service.cc:1467] Processing CreateTablet for tablet 1444986ee92e4c70b16cb30d0d906157 (DEFAULT_TABLE table=test-table [id=14cab0de4c1d442b946cc3eb9acf30a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:09.268462 9109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1444986ee92e4c70b16cb30d0d906157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:09.271867 9560 ts_tablet_manager.cc:1918] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:09.272019 9560 log.cc:1196] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/08df1bda83254e71bb642c7eb18b58ab
I20260619 13:01:09.272581 9560 ts_tablet_manager.cc:1939] T 08df1bda83254e71bb642c7eb18b58ab P 261db81c10d545a99baeb865951226ac: Deleting consensus metadata
I20260619 13:01:09.273023 9247 tablet_service.cc:1467] Processing CreateTablet for tablet 1444986ee92e4c70b16cb30d0d906157 (DEFAULT_TABLE table=test-table [id=14cab0de4c1d442b946cc3eb9acf30a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:09.273470 9247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1444986ee92e4c70b16cb30d0d906157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:09.273542 8956 catalog_manager.cc:4841] TS 261db81c10d545a99baeb865951226ac (127.1.87.67:37633): tablet 08df1bda83254e71bb642c7eb18b58ab (table test-table [id=3e89dc92f225469b94860befa2989112]) successfully deleted
I20260619 13:01:09.274804 9518 tablet_bootstrap.cc:492] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: Bootstrap starting.
I20260619 13:01:09.276186 9518 tablet_bootstrap.cc:654] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:09.278162 9518 tablet_bootstrap.cc:492] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: No bootstrap required, opened a new log
I20260619 13:01:09.278283 9518 ts_tablet_manager.cc:1397] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:01:09.278829 9518 raft_consensus.cc:348] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.279004 9518 raft_consensus.cc:374] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:09.279058 9518 raft_consensus.cc:729] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b28a41b03e41ee864e404b5c45f67e, State: Initialized, Role: FOLLOWER
I20260619 13:01:09.279285 9518 consensus_queue.cc:260] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.279913 9517 tablet_bootstrap.cc:492] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: Bootstrap starting.
I20260619 13:01:09.281446 9517 tablet_bootstrap.cc:654] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:09.282333 9518 ts_tablet_manager.cc:1428] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 13:01:09.283201 9516 tablet_bootstrap.cc:492] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: Bootstrap starting.
I20260619 13:01:09.284916 9516 tablet_bootstrap.cc:654] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:09.287509 9503 raft_consensus.cc:482] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:09.287683 9503 raft_consensus.cc:504] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.288304 9503 leader_election.cc:288] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:09.288676 9268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444986ee92e4c70b16cb30d0d906157" candidate_uuid: "a0b28a41b03e41ee864e404b5c45f67e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf31521f80646bbadede2ce9289bb11" is_pre_election: true
I20260619 13:01:09.288947 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444986ee92e4c70b16cb30d0d906157" candidate_uuid: "a0b28a41b03e41ee864e404b5c45f67e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac" is_pre_election: true
W20260619 13:01:09.289695 9062 leader_election.cc:341] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225): Illegal state: must be running to vote when last-logged opid is not known
W20260619 13:01:09.289996 9059 leader_election.cc:341] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 261db81c10d545a99baeb865951226ac (127.1.87.67:37633): Illegal state: must be running to vote when last-logged opid is not known
I20260619 13:01:09.290139 9059 leader_election.cc:302] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0b28a41b03e41ee864e404b5c45f67e; no voters: 261db81c10d545a99baeb865951226ac, aaf31521f80646bbadede2ce9289bb11
I20260619 13:01:09.290730 9503 raft_consensus.cc:2726] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 13:01:09.291100 8959 catalog_manager.cc:4841] TS aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225): tablet 08df1bda83254e71bb642c7eb18b58ab (table test-table [id=3e89dc92f225469b94860befa2989112]) successfully deleted
W20260619 13:01:09.291610 8959 catalog_manager.cc:4808] TS aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225): delete failed for tablet 08df1bda83254e71bb642c7eb18b58ab because the tablet was not found. No further retry: Not found: Tablet not found: 08df1bda83254e71bb642c7eb18b58ab
I20260619 13:01:09.294867 9517 tablet_bootstrap.cc:492] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: No bootstrap required, opened a new log
I20260619 13:01:09.295042 9517 ts_tablet_manager.cc:1397] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20260619 13:01:09.295838 9517 raft_consensus.cc:348] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.296090 9517 raft_consensus.cc:374] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:09.296172 9517 raft_consensus.cc:729] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf31521f80646bbadede2ce9289bb11, State: Initialized, Role: FOLLOWER
I20260619 13:01:09.296466 9516 tablet_bootstrap.cc:492] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: No bootstrap required, opened a new log
I20260619 13:01:09.296447 9517 consensus_queue.cc:260] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.296641 9516 ts_tablet_manager.cc:1397] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260619 13:01:09.297115 9517 ts_tablet_manager.cc:1428] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:01:09.297359 9516 raft_consensus.cc:348] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.297614 9516 raft_consensus.cc:374] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:09.297694 9516 raft_consensus.cc:729] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261db81c10d545a99baeb865951226ac, State: Initialized, Role: FOLLOWER
I20260619 13:01:09.297960 9516 consensus_queue.cc:260] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.298484 9516 ts_tablet_manager.cc:1428] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 13:01:09.326215 9488 raft_consensus.cc:482] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:09.326416 9488 raft_consensus.cc:504] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.327178 9488 leader_election.cc:288] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:09.327740 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444986ee92e4c70b16cb30d0d906157" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac" is_pre_election: true
I20260619 13:01:09.327759 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444986ee92e4c70b16cb30d0d906157" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e" is_pre_election: true
I20260619 13:01:09.327978 9412 raft_consensus.cc:2454] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 0.
I20260619 13:01:09.328012 9134 raft_consensus.cc:2454] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 0.
I20260619 13:01:09.328352 9199 leader_election.cc:302] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261db81c10d545a99baeb865951226ac, aaf31521f80646bbadede2ce9289bb11; no voters:
I20260619 13:01:09.328616 9488 raft_consensus.cc:2781] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:01:09.328749 9488 raft_consensus.cc:482] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:01:09.328860 9488 raft_consensus.cc:3037] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:09.330947 9488 raft_consensus.cc:504] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.331558 9488 leader_election.cc:288] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 election: Requested vote from peers a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991), 261db81c10d545a99baeb865951226ac (127.1.87.67:37633)
I20260619 13:01:09.331976 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444986ee92e4c70b16cb30d0d906157" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b28a41b03e41ee864e404b5c45f67e"
I20260619 13:01:09.332023 9412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444986ee92e4c70b16cb30d0d906157" candidate_uuid: "aaf31521f80646bbadede2ce9289bb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261db81c10d545a99baeb865951226ac"
I20260619 13:01:09.332185 9134 raft_consensus.cc:3037] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:09.332229 9412 raft_consensus.cc:3037] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:09.334169 9412 raft_consensus.cc:2454] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 1.
I20260619 13:01:09.334623 9199 leader_election.cc:302] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261db81c10d545a99baeb865951226ac, aaf31521f80646bbadede2ce9289bb11; no voters:
I20260619 13:01:09.334889 9488 raft_consensus.cc:2781] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:09.335062 9488 raft_consensus.cc:686] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 1 LEADER]: Becoming Leader. State: Replica: aaf31521f80646bbadede2ce9289bb11, State: Running, Role: LEADER
I20260619 13:01:09.335183 9134 raft_consensus.cc:2454] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf31521f80646bbadede2ce9289bb11 in term 1.
I20260619 13:01:09.335397 9488 consensus_queue.cc:237] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } }
I20260619 13:01:09.338341 8970 catalog_manager.cc:5495] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf31521f80646bbadede2ce9289bb11 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "aaf31521f80646bbadede2ce9289bb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf31521f80646bbadede2ce9289bb11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 } health_report { overall_health: UNKNOWN } } }
I20260619 13:01:09.371510 9134 raft_consensus.cc:1261] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Refusing update from remote peer aaf31521f80646bbadede2ce9289bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:09.371531 9412 raft_consensus.cc:1261] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Refusing update from remote peer aaf31521f80646bbadede2ce9289bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:09.372059 9495 consensus_queue.cc:1035] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b28a41b03e41ee864e404b5c45f67e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:09.372514 9495 consensus_queue.cc:1035] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "261db81c10d545a99baeb865951226ac" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 37633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:09.417027 8970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:09.417373 8970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:09.420274 8970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9028c1ef37e042e1801476ab421d0f78: Sending DeleteTablet for 3 replicas of tablet 1444986ee92e4c70b16cb30d0d906157
I20260619 13:01:09.421092 9109 tablet_service.cc:1514] Processing DeleteTablet for tablet 1444986ee92e4c70b16cb30d0d906157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:47984
I20260619 13:01:09.421106 9247 tablet_service.cc:1514] Processing DeleteTablet for tablet 1444986ee92e4c70b16cb30d0d906157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:40774
I20260619 13:01:09.421212 9386 tablet_service.cc:1514] Processing DeleteTablet for tablet 1444986ee92e4c70b16cb30d0d906157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:09 UTC) from {username='oryx', principal='oryx/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:53124
I20260619 13:01:09.421389 9558 tablet_replica.cc:330] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: stopping tablet replica
I20260619 13:01:09.421490 9559 tablet_replica.cc:330] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: stopping tablet replica
I20260619 13:01:09.421485 9560 tablet_replica.cc:330] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: stopping tablet replica
I20260619 13:01:09.421566 9558 raft_consensus.cc:2229] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:09.421670 9559 raft_consensus.cc:2229] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:01:09.421665 9560 raft_consensus.cc:2229] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:09.421752 9558 raft_consensus.cc:2258] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:09.421850 9560 raft_consensus.cc:2258] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:09.421942 9559 raft_consensus.cc:2258] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:09.422902 9560 ts_tablet_manager.cc:1905] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:09.422902 9558 ts_tablet_manager.cc:1905] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:09.422921 9559 ts_tablet_manager.cc:1905] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:09.423893 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 9043
I20260619 13:01:09.429764 9560 ts_tablet_manager.cc:1918] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:01:09.429764 9558 ts_tablet_manager.cc:1918] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:01:09.429900 9560 log.cc:1196] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/1444986ee92e4c70b16cb30d0d906157
I20260619 13:01:09.430058 9558 log.cc:1196] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/1444986ee92e4c70b16cb30d0d906157
I20260619 13:01:09.430552 9560 ts_tablet_manager.cc:1939] T 1444986ee92e4c70b16cb30d0d906157 P 261db81c10d545a99baeb865951226ac: Deleting consensus metadata
I20260619 13:01:09.430688 9558 ts_tablet_manager.cc:1939] T 1444986ee92e4c70b16cb30d0d906157 P a0b28a41b03e41ee864e404b5c45f67e: Deleting consensus metadata
I20260619 13:01:09.431598 8956 catalog_manager.cc:4841] TS 261db81c10d545a99baeb865951226ac (127.1.87.67:37633): tablet 1444986ee92e4c70b16cb30d0d906157 (table test-table [id=14cab0de4c1d442b946cc3eb9acf30a7]) successfully deleted
I20260619 13:01:09.431777 8959 catalog_manager.cc:4841] TS a0b28a41b03e41ee864e404b5c45f67e (127.1.87.65:38991): tablet 1444986ee92e4c70b16cb30d0d906157 (table test-table [id=14cab0de4c1d442b946cc3eb9acf30a7]) successfully deleted
I20260619 13:01:09.432380 9559 ts_tablet_manager.cc:1918] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:01:09.432546 9559 log.cc:1196] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/1444986ee92e4c70b16cb30d0d906157
I20260619 13:01:09.432912 9559 ts_tablet_manager.cc:1939] T 1444986ee92e4c70b16cb30d0d906157 P aaf31521f80646bbadede2ce9289bb11: Deleting consensus metadata
I20260619 13:01:09.433766 8959 catalog_manager.cc:4841] TS aaf31521f80646bbadede2ce9289bb11 (127.1.87.66:38225): tablet 1444986ee92e4c70b16cb30d0d906157 (table test-table [id=14cab0de4c1d442b946cc3eb9acf30a7]) successfully deleted
Jun 19 13:01:09 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.65@KRBTEST.COM
I20260619 13:01:09.445237 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:09.424865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.1.87.65:38991)
0619 13:01:09.425041 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:09.425049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:09.432234 (+ 7185us) server_negotiation.cc:407] Connection header received
0619 13:01:09.432450 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:09.432454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:09.432548 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:09.432678 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:09.433850 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.434555 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.435837 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.436237 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.436370 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:09.440136 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:09.440149 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:09.440164 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:09.440192 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:09.442650 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:09.443272 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:09.443280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:09.443282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:09.443446 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:09.443802 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:09.443807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:09.443810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:09.444132 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:09.444463 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:09.444688 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 13:01:09.444897 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:01:09.445873 9174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:09.566114 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 9043
I20260619 13:01:09.588413 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 9183
Jun 19 13:01:09 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.66@KRBTEST.COM
I20260619 13:01:09.603233 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:09.589152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.1.87.66:38225)
0619 13:01:09.589369 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:09.589376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:09.589476 (+ 100us) server_negotiation.cc:407] Connection header received
0619 13:01:09.589689 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:09.589693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:09.589770 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:09.589876 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:09.590962 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.591742 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.592944 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.593329 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.593465 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:09.597496 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:09.597510 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:09.597527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:09.597559 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:09.600527 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:09.601223 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:09.601230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:09.601233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:09.601351 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:09.601711 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:09.601717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:09.601720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:09.602065 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:09.602372 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:09.602684 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 13:01:09.602909 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:01:09.603996 9313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:09.736238 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 9183
I20260619 13:01:09.758453 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 9322
Jun 19 13:01:09 dist-test-slave-tpsm krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.87.67@KRBTEST.COM
I20260619 13:01:09.773046 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:09.759242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.1.87.67:37633)
0619 13:01:09.759809 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:09.759813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:09.759830 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:01:09.759879 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:09.759883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:09.759944 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:09.760035 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:09.761310 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.762052 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.763484 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:09.763908 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:09.764005 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:09.768115 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:09.768126 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:09.768141 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:09.768170 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:09.770779 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:09.771389 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:09.771396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:09.771398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:09.771508 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:09.771818 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:09.771821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:09.771823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:09.772124 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:09.772336 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:09.772545 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 13:01:09.772693 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":350,"threads_started":1}
I20260619 13:01:09.773649 9452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:09.867511 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 9322
I20260619 13:01:09.884958 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 8940
I20260619 13:01:09.886075 9012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:10.002727 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 8940
2026-06-19T13:01:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5758 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:01:10 dist-test-slave-tpsm krb5kdc[9595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:01:10 dist-test-slave-tpsm krb5kdc[9595](info): set up 2 sockets
Jun 19 13:01:10 dist-test-slave-tpsm krb5kdc[9595](info): commencing operation
krb5kdc: starting...
W20260619 13:01:11.096422 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:01:11 dist-test-slave-tpsm krb5kdc[9595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:01:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:01:11.302330 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:35167
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:35167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:11.617624 9608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:11.617899 9608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:11.618053 9608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:11.625205 9608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:01:11.625309 9608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:11.625391 9608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:11.625495 9608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:01:11.625576 9608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:01:11.635218 9608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:35167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:35167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:11.636531 9608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:11.637782 9608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:11.649062 9613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:11.649013 9614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:11.649086 9616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:11.649960 9608 server_base.cc:1034] running on GCE node
I20260619 13:01:11.650691 9608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:11.652079 9608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:11.653316 9608 hybrid_clock.cc:648] HybridClock initialized: now 1781874071653264 us; error 58 us; skew 500 ppm
I20260619 13:01:11.653656 9608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T13:01:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1712 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:01:11 dist-test-slave-tpsm krb5kdc[9622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:01:11 dist-test-slave-tpsm krb5kdc[9622](info): set up 2 sockets
Jun 19 13:01:11 dist-test-slave-tpsm krb5kdc[9622](info): commencing operation
krb5kdc: starting...
W20260619 13:01:12.832402 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:01:12 dist-test-slave-tpsm krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:01:13.046816 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37527
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:37527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:13.361879 9635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:13.362159 9635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:13.362363 9635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:13.369613 9635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:01:13.369719 9635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:13.369830 9635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:13.369931 9635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:01:13.370015 9635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:01:13.379946 9635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:37527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:13.381235 9635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:13.382517 9635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:13.394110 9641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:13.394141 9640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:13.394316 9643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:13.394949 9635 server_base.cc:1034] running on GCE node
I20260619 13:01:13.395825 9635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:13.397171 9635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:13.398396 9635 hybrid_clock.cc:648] HybridClock initialized: now 1781874073398383 us; error 54 us; skew 500 ppm
I20260619 13:01:13.398722 9635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:13 dist-test-slave-tpsm krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:13.401654 9635 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:01:13.403053 9635 webserver.cc:460] Webserver started at http://127.1.87.126:43991/ using document root <none> and password file <none>
I20260619 13:01:13.403779 9635 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:13.403918 9635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:13.404222 9635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:13.407032 9635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "ffd28d2ac25541aebfd1bb8552efd4d9"
format_stamp: "Formatted at 2026-06-19 13:01:13 on dist-test-slave-tpsm"
server_key: "5c1365bf4759f7443ab7c5abcdd74cf4"
server_key_iv: "f320275ea0d081c575df2ce7425874a9"
server_key_version: "encryptionkey@0"
I20260619 13:01:13.407846 9635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "ffd28d2ac25541aebfd1bb8552efd4d9"
format_stamp: "Formatted at 2026-06-19 13:01:13 on dist-test-slave-tpsm"
server_key: "5c1365bf4759f7443ab7c5abcdd74cf4"
server_key_iv: "f320275ea0d081c575df2ce7425874a9"
server_key_version: "encryptionkey@0"
I20260619 13:01:13.412546 9635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 13:01:13.415694 9650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:13.416886 9635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 13:01:13.417063 9635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "ffd28d2ac25541aebfd1bb8552efd4d9"
format_stamp: "Formatted at 2026-06-19 13:01:13 on dist-test-slave-tpsm"
server_key: "5c1365bf4759f7443ab7c5abcdd74cf4"
server_key_iv: "f320275ea0d081c575df2ce7425874a9"
server_key_version: "encryptionkey@0"
I20260619 13:01:13.417254 9635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:13.436070 9635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:13.439888 9635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:13.440106 9635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:13.457926 9635 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:37527
I20260619 13:01:13.457965 9701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:37527 every 8 connection(s)
I20260619 13:01:13.459272 9635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:01:13.462847 9702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:13.467005 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 9635
I20260619 13:01:13.467195 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:01:13.467849 1373 external_mini_cluster.cc:1428] Setting key 76394f956d73dd6e109def81e7fd66de
I20260619 13:01:13.474572 9702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: Bootstrap starting.
I20260619 13:01:13.478036 9702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:13.479072 9702 log.cc:824] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:13.482012 9702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: No bootstrap required, opened a new log
Jun 19 13:01:13 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:01:13.492549 9702 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } }
I20260619 13:01:13.492965 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:13.469755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.1.87.126:37527)
0619 13:01:13.470577 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:13.470593 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:13.470639 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:01:13.473094 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:13.473120 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:13.473576 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:13.473970 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:13.475378 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:13.477352 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:13.478752 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:13.479849 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:13.479974 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:13.484525 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:13.484536 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:13.484560 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:13.484602 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:13.488586 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:13.489296 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:13.489306 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:13.489319 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:13.489517 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:13.489852 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:13.489857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:13.489861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:13.490573 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:13.490890 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:13.491234 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 13:01:13.491688 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":341,"threads_started":1}
I20260619 13:01:13.492986 9702 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:13.493108 9702 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd28d2ac25541aebfd1bb8552efd4d9, State: Initialized, Role: FOLLOWER
I20260619 13:01:13.493686 9702 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } }
I20260619 13:01:13.493938 9702 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:01:13.494078 9702 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:01:13.494278 9702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:13.497208 9702 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } }
I20260619 13:01:13.497629 9702 leader_election.cc:302] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd28d2ac25541aebfd1bb8552efd4d9; no voters:
I20260619 13:01:13.498117 9702 leader_election.cc:288] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:01:13.498268 9707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:13.498940 9707 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 1 LEADER]: Becoming Leader. State: Replica: ffd28d2ac25541aebfd1bb8552efd4d9, State: Running, Role: LEADER
I20260619 13:01:13.499493 9707 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } }
I20260619 13:01:13.500155 9702 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:01:13.502665 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } } }
I20260619 13:01:13.503199 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:13.502785 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd28d2ac25541aebfd1bb8552efd4d9. Latest consensus state: current_term: 1 leader_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } } }
I20260619 13:01:13.503811 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:13.506032 9716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:01:13.510938 9716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:01:13.519398 9716 catalog_manager.cc:1347] Generated new cluster ID: a677ded7e3bc401fbfdd459e44970b6b
I20260619 13:01:13.519512 9716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:01:13.547998 9716 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:01:13.549368 9716 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:01:13.558337 9716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: Generated new TSK 0
I20260619 13:01:13.559188 9716 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:01:13.617151 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37527
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:13.941581 9728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:13.941901 9728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:13.942073 9728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:13.949455 9728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:13.949568 9728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:13.949795 9728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:01:13.959584 9728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:13.960942 9728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:13.962242 9728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:13.975548 9734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:13.975656 9733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:13.975569 9736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:13.980959 9728 server_base.cc:1034] running on GCE node
I20260619 13:01:13.981643 9728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:13.982611 9728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:13.983822 9728 hybrid_clock.cc:648] HybridClock initialized: now 1781874073983776 us; error 34 us; skew 500 ppm
I20260619 13:01:13.984164 9728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:13 dist-test-slave-tpsm krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:13.988163 9728 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:01:13.989861 9728 webserver.cc:460] Webserver started at http://127.1.87.65:42607/ using document root <none> and password file <none>
I20260619 13:01:13.990577 9728 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:13.990732 9728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:13.991039 9728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:13.994302 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "c24fab2e749542f5bd65d71d2d282b14"
format_stamp: "Formatted at 2026-06-19 13:01:13 on dist-test-slave-tpsm"
server_key: "c1068be60a2874405ebfe7a9f3e58fb1"
server_key_iv: "f7a1a0baaf45ea80ae51a544826e3f1b"
server_key_version: "encryptionkey@0"
I20260619 13:01:13.995065 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "c24fab2e749542f5bd65d71d2d282b14"
format_stamp: "Formatted at 2026-06-19 13:01:13 on dist-test-slave-tpsm"
server_key: "c1068be60a2874405ebfe7a9f3e58fb1"
server_key_iv: "f7a1a0baaf45ea80ae51a544826e3f1b"
server_key_version: "encryptionkey@0"
I20260619 13:01:14.000053 9728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:01:14.003790 9743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:14.005285 9728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:01:14.005465 9728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "c24fab2e749542f5bd65d71d2d282b14"
format_stamp: "Formatted at 2026-06-19 13:01:13 on dist-test-slave-tpsm"
server_key: "c1068be60a2874405ebfe7a9f3e58fb1"
server_key_iv: "f7a1a0baaf45ea80ae51a544826e3f1b"
server_key_version: "encryptionkey@0"
I20260619 13:01:14.005681 9728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:14.042265 9728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:14.046659 9728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:14.046969 9728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:14.047948 9728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:14.049322 9728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:14.049427 9728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:14.049587 9728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:14.049660 9728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:14.083190 9728 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:46001
I20260619 13:01:14.083230 9855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:46001 every 8 connection(s)
I20260619 13:01:14.084479 9728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:01:14.088666 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 9728
I20260619 13:01:14.088860 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:01:14.089694 1373 external_mini_cluster.cc:1428] Setting key eb2ca1cc20025e6a7495cd83d9cfa59b
Jun 19 13:01:14 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:01:14.107635 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:14.087082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:41459 (local address 127.1.87.126:37527)
0619 13:01:14.087554 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:14.087559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:14.088857 (+ 1298us) server_negotiation.cc:407] Connection header received
0619 13:01:14.090990 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:14.090995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:14.091095 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:14.091214 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:14.093255 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:14.093958 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:14.096427 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:14.096826 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:14.096922 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:14.101487 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:14.101495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:14.101501 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:14.101567 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:14.104433 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:14.105315 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:14.105321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:14.105325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:14.105451 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:14.105876 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:14.105881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:14.105885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:14.106159 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:14.106313 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:14.106992 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 13:01:14.107210 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":259,"threads_started":1}
I20260619 13:01:14.108686 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:14.087686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:41459)
0619 13:01:14.088496 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:14.088547 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:14.090746 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:14.091534 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:14.091543 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:14.092244 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:14.093043 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:14.093072 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:14.094136 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:14.094144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:14.096265 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:14.096284 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:14.097057 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:14.097066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:14.097401 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:14.098161 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:14.098201 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:14.101081 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:14.104577 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:14.104587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:14.104605 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:14.105184 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:14.105563 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:14.105568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:14.105572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:14.105757 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:14.106286 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:14.106294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:14.106634 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 13:01:14.106892 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 13:01:14.107452 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":299,"threads_started":1}
I20260619 13:01:14.110484 9856 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37527
I20260619 13:01:14.110785 9856 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:14.111550 9856 heartbeater.cc:511] Master 127.1.87.126:37527 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
I20260619 13:01:14.113693 9667 ts_manager.cc:194] Registered new tserver with Master: c24fab2e749542f5bd65d71d2d282b14 (127.1.87.65:46001)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
I20260619 13:01:14.115159 9667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:41459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:01:14.138895 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37527
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:14.461926 9862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:14.462209 9862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:14.462374 9862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:14.472664 9862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:14.472818 9862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:14.472992 9862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:01:14.483371 9862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:14.484735 9862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:14.485965 9862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:14.499094 9867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:14.499156 9870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:14.499181 9868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:14.499935 9862 server_base.cc:1034] running on GCE node
I20260619 13:01:14.500609 9862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:14.501665 9862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:14.502924 9862 hybrid_clock.cc:648] HybridClock initialized: now 1781874074502869 us; error 55 us; skew 500 ppm
I20260619 13:01:14.503263 9862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:14 dist-test-slave-tpsm krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874074, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:14.506441 9862 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:01:14.507995 9862 webserver.cc:460] Webserver started at http://127.1.87.66:35515/ using document root <none> and password file <none>
I20260619 13:01:14.508656 9862 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:14.508782 9862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:14.509091 9862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:14.512216 9862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "a26c5d8747b3463890c4b4e50739d91f"
format_stamp: "Formatted at 2026-06-19 13:01:14 on dist-test-slave-tpsm"
server_key: "142d2a583d250c36836e212366abc89d"
server_key_iv: "5749ccdd84a3f65a622f3b55020b25a1"
server_key_version: "encryptionkey@0"
I20260619 13:01:14.512924 9862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "a26c5d8747b3463890c4b4e50739d91f"
format_stamp: "Formatted at 2026-06-19 13:01:14 on dist-test-slave-tpsm"
server_key: "142d2a583d250c36836e212366abc89d"
server_key_iv: "5749ccdd84a3f65a622f3b55020b25a1"
server_key_version: "encryptionkey@0"
I20260619 13:01:14.518893 9862 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 13:01:14.522390 9877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:14.523856 9862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:01:14.524061 9862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "a26c5d8747b3463890c4b4e50739d91f"
format_stamp: "Formatted at 2026-06-19 13:01:14 on dist-test-slave-tpsm"
server_key: "142d2a583d250c36836e212366abc89d"
server_key_iv: "5749ccdd84a3f65a622f3b55020b25a1"
server_key_version: "encryptionkey@0"
I20260619 13:01:14.524300 9862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:14.561311 9862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:14.565685 9862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:14.565984 9862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:14.566975 9862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:14.568368 9862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:14.568473 9862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:14.568634 9862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:14.568706 9862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:14.602684 9862 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:41863
I20260619 13:01:14.602773 9989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:41863 every 8 connection(s)
I20260619 13:01:14.604883 9862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:01:14.610764 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 9862
I20260619 13:01:14.610951 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:01:14.611474 1373 external_mini_cluster.cc:1428] Setting key 3e070072170f261ca9440b094c81e2b7
Jun 19 13:01:14 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874074, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:01:14.626223 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:14.607472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:51737 (local address 127.1.87.126:37527)
0619 13:01:14.607652 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:14.607658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:14.608798 (+ 1140us) server_negotiation.cc:407] Connection header received
0619 13:01:14.610777 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:14.610782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:14.610881 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:14.610975 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:14.612686 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:14.613503 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:14.615011 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:14.615503 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:14.615645 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:14.619252 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:14.619256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:14.619260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:14.619291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:14.622839 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:14.623642 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:14.623651 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:14.623657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:14.623811 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:14.624264 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:14.624271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:14.624277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:14.624652 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:14.624812 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:14.625507 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 13:01:14.625762 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:01:14.626928 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:14.607950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:51737)
0619 13:01:14.608629 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:14.608667 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:14.610505 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:14.611215 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:14.611223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:14.611833 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:14.612514 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:14.612533 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:14.613645 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:14.613651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:14.614859 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:14.614871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:14.615729 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:14.615736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:14.615958 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:14.616672 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:14.616699 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:14.619014 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:14.623001 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:14.623008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:14.623022 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:14.623524 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:14.623937 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:14.623940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:14.623943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:14.624132 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:14.624778 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:14.624785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:14.625134 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 13:01:14.625365 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 13:01:14.625753 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":271,"threads_started":1}
I20260619 13:01:14.628485 9990 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37527
I20260619 13:01:14.628840 9990 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:14.629667 9990 heartbeater.cc:511] Master 127.1.87.126:37527 requested a full tablet report, sending...
I20260619 13:01:14.631136 9667 ts_manager.cc:194] Registered new tserver with Master: a26c5d8747b3463890c4b4e50739d91f (127.1.87.66:41863)
I20260619 13:01:14.632002 9667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:51737
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:01:14.662253 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37527
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:14.987167 9995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:14.987599 9995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:14.987864 9995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:14.995941 9995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:14.996078 9995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:14.996270 9995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:01:15.006088 9995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:15.007414 9995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:15.008680 9995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:15.021740 10003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:15.021814 10001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:15.022215 10000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:15.024760 9995 server_base.cc:1034] running on GCE node
I20260619 13:01:15.025341 9995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:15.026242 9995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:15.027423 9995 hybrid_clock.cc:648] HybridClock initialized: now 1781874075027384 us; error 29 us; skew 500 ppm
I20260619 13:01:15.027784 9995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:15 dist-test-slave-tpsm krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:15.030830 9995 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:01:15.032404 9995 webserver.cc:460] Webserver started at http://127.1.87.67:34817/ using document root <none> and password file <none>
I20260619 13:01:15.033085 9995 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:15.033229 9995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:15.033529 9995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:15.036370 9995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "737ae9740cf442a186be113ddedc2d16"
format_stamp: "Formatted at 2026-06-19 13:01:15 on dist-test-slave-tpsm"
server_key: "b7b3c6d4f9ab08c27120db7dd000bca5"
server_key_iv: "4aa8478c4f1a14e4708cb122d9a547db"
server_key_version: "encryptionkey@0"
I20260619 13:01:15.037117 9995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "737ae9740cf442a186be113ddedc2d16"
format_stamp: "Formatted at 2026-06-19 13:01:15 on dist-test-slave-tpsm"
server_key: "b7b3c6d4f9ab08c27120db7dd000bca5"
server_key_iv: "4aa8478c4f1a14e4708cb122d9a547db"
server_key_version: "encryptionkey@0"
I20260619 13:01:15.042066 9995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:01:15.045686 10010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:15.046921 9995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 13:01:15.047099 9995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "737ae9740cf442a186be113ddedc2d16"
format_stamp: "Formatted at 2026-06-19 13:01:15 on dist-test-slave-tpsm"
server_key: "b7b3c6d4f9ab08c27120db7dd000bca5"
server_key_iv: "4aa8478c4f1a14e4708cb122d9a547db"
server_key_version: "encryptionkey@0"
I20260619 13:01:15.047313 9995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:15.063381 9995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:15.067770 9995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:15.068171 9995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:15.069186 9995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:15.070574 9995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:15.070688 9995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:15.070847 9995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:15.070930 9995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:15.106647 9995 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:35673
I20260619 13:01:15.106691 10122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:35673 every 8 connection(s)
I20260619 13:01:15.107978 9995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:01:15.113646 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 9995
I20260619 13:01:15.113842 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:01:15.114266 1373 external_mini_cluster.cc:1428] Setting key 9d99ecfed38122e85b0af157fa2a968f
I20260619 13:01:15.117928 9856 heartbeater.cc:503] Master 127.1.87.126:37527 was elected leader, sending a full tablet report...
Jun 19 13:01:15 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:01:15.131647 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:15.110478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:37249 (local address 127.1.87.126:37527)
0619 13:01:15.110649 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:15.110654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:15.112163 (+ 1509us) server_negotiation.cc:407] Connection header received
0619 13:01:15.113996 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:15.114002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:15.114109 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:15.114218 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:15.116303 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.117248 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.119142 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.119782 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.119883 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:15.123388 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:15.123393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:15.123397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:15.123428 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:15.126738 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:15.128066 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:15.128072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:15.128075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:15.128282 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:15.128844 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:15.128851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:15.128854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:15.129237 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:15.129423 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:15.130946 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0619 13:01:15.131136 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:01:15.133148 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:15.110951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:37249)
0619 13:01:15.111987 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:15.112026 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:15.113674 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:15.114473 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:15.114481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:15.115135 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:15.116116 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:15.116136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.117432 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.117437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:15.118974 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:15.118987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.119982 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.119988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:15.120206 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:15.120881 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:15.120904 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:15.123109 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:15.126965 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:15.126979 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:15.126995 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:15.127885 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:15.128457 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:15.128462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:15.128466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:15.128716 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:15.129518 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:15.129530 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:15.130238 (+ 708us) client_negotiation.cc:769] Sending connection context
0619 13:01:15.130794 (+ 556us) client_negotiation.cc:240] Negotiation successful
0619 13:01:15.131287 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":445,"threads_started":1}
I20260619 13:01:15.135037 10123 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37527
I20260619 13:01:15.135434 10123 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:15.136392 10123 heartbeater.cc:511] Master 127.1.87.126:37527 requested a full tablet report, sending...
I20260619 13:01:15.137972 9667 ts_manager.cc:194] Registered new tserver with Master: 737ae9740cf442a186be113ddedc2d16 (127.1.87.67:35673)
I20260619 13:01:15.138792 9667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:37249
I20260619 13:01:15.142936 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab.
Entry for principal oryx/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab.
WARNING: no policy specified for oryx/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab.
Entry for principal oryx/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab.
WARNING: no policy specified for oryx/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab.
Entry for principal oryx/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab.
I20260619 13:01:15.281983 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 9728
Jun 19 13:01:15 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:01:15.300544 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:15.283202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.1.87.65:46001)
0619 13:01:15.284576 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:15.284584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:15.284614 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:01:15.284680 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:15.284686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:15.284911 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:15.285061 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:15.286444 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.287621 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.289169 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.290050 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.290207 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:15.294674 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:15.294687 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:15.294703 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:15.294740 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:15.297647 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:15.298398 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:15.298406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:15.298408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:15.298527 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:15.298901 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:15.298908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:15.298911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:15.299250 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:15.299594 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:15.299835 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 13:01:15.300104 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":1104,"threads_started":1}
I20260619 13:01:15.301913 9851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:15.383661 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 9728
I20260619 13:01:15.399528 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 9862
Jun 19 13:01:15 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:01:15.414337 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:15.400383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.1.87.66:41863)
0619 13:01:15.400959 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:15.400966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:15.400990 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:01:15.401048 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:15.401054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:15.401244 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:15.401370 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:15.402457 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.403563 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.404748 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.405271 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.405416 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:15.409275 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:15.409286 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:15.409299 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:15.409335 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:15.412025 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:15.412617 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:15.412623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:15.412626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:15.412737 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:15.413021 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:15.413024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:15.413025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:15.413322 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:15.413531 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:15.413855 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 13:01:15.414034 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":291,"threads_started":1}
I20260619 13:01:15.415627 9985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:15.496341 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 9862
I20260619 13:01:15.513697 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 9995
Jun 19 13:01:15 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:01:15.529750 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:15.514569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.1.87.67:35673)
0619 13:01:15.515186 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:15.515194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:15.515224 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:01:15.515284 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:15.515290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:15.515566 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:15.515698 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:15.516955 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.517976 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.519188 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:15.519862 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:15.519973 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:15.524407 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:15.524422 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:15.524437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:15.524472 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:15.527511 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:15.528122 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:15.528128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:15.528130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:15.528236 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:15.528539 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:15.528542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:15.528544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:15.528850 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:15.529069 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:15.529289 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 13:01:15.529466 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":307,"threads_started":1}
I20260619 13:01:15.530936 10118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:15.613262 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 9995
I20260619 13:01:15.629012 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 9635
I20260619 13:01:15.630288 9697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:15.731868 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 9635
I20260619 13:01:15.752908 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37527
--webserver_interface=127.1.87.126
--webserver_port=43991
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:37527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:16.066794 10140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:16.067060 10140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:16.067227 10140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:16.074218 10140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:01:16.074316 10140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:16.074422 10140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:16.074527 10140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:01:16.074612 10140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:01:16.084129 10140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:37527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=43991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:16.085389 10140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:16.086638 10140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:16.097960 10146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:16.097929 10145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:16.097914 10148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:16.098753 10140 server_base.cc:1034] running on GCE node
I20260619 13:01:16.099545 10140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:16.100808 10140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:16.102043 10140 hybrid_clock.cc:648] HybridClock initialized: now 1781874076101996 us; error 66 us; skew 500 ppm
I20260619 13:01:16.102384 10140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:16 dist-test-slave-tpsm krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874076, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:16.105396 10140 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:01:16.106729 10140 webserver.cc:460] Webserver started at http://127.1.87.126:43991/ using document root <none> and password file <none>
I20260619 13:01:16.107477 10140 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:16.107628 10140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:16.112690 10140 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 13:01:16.115439 10155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:16.116671 10140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:01:16.116842 10140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "ffd28d2ac25541aebfd1bb8552efd4d9"
format_stamp: "Formatted at 2026-06-19 13:01:13 on dist-test-slave-tpsm"
server_key: "5c1365bf4759f7443ab7c5abcdd74cf4"
server_key_iv: "f320275ea0d081c575df2ce7425874a9"
server_key_version: "encryptionkey@0"
I20260619 13:01:16.117746 10140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:16.145280 10140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:16.149288 10140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:16.149564 10140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:16.167896 10140 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:37527
I20260619 13:01:16.167932 10206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:37527 every 8 connection(s)
I20260619 13:01:16.169375 10140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:01:16.174664 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 10140
I20260619 13:01:16.175611 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:46001
--local_ip_for_outbound_sockets=127.1.87.65
--tserver_master_addrs=127.1.87.126:37527
--webserver_port=42607
--webserver_interface=127.1.87.65
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.87.65
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:01:16.176620 10207 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:01:16.181752 10207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: Bootstrap starting.
I20260619 13:01:16.200556 10207 log.cc:824] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:16.209472 10207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:01:16.210016 10207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: Bootstrap complete.
I20260619 13:01:16.219048 10207 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } }
I20260619 13:01:16.219980 10207 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd28d2ac25541aebfd1bb8552efd4d9, State: Initialized, Role: FOLLOWER
I20260619 13:01:16.220736 10207 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } }
I20260619 13:01:16.221009 10207 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:01:16.221123 10207 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:01:16.221299 10207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:01:16.225345 10207 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } }
I20260619 13:01:16.225880 10207 leader_election.cc:302] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd28d2ac25541aebfd1bb8552efd4d9; no voters:
I20260619 13:01:16.226516 10207 leader_election.cc:288] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:01:16.226641 10211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:01:16.227905 10211 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [term 2 LEADER]: Becoming Leader. State: Replica: ffd28d2ac25541aebfd1bb8552efd4d9, State: Running, Role: LEADER
I20260619 13:01:16.228524 10211 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } }
I20260619 13:01:16.228798 10207 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:01:16.231334 10213 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd28d2ac25541aebfd1bb8552efd4d9. Latest consensus state: current_term: 2 leader_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } } }
I20260619 13:01:16.231680 10213 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:16.231419 10212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd28d2ac25541aebfd1bb8552efd4d9" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37527 } } }
I20260619 13:01:16.231856 10212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:16.244794 10218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:01:16.250749 10218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:01:16.252189 10218 catalog_manager.cc:1259] Loaded cluster ID: a677ded7e3bc401fbfdd459e44970b6b
I20260619 13:01:16.252344 10218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:01:16.256289 10218 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:01:16.257695 10218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffd28d2ac25541aebfd1bb8552efd4d9: Loaded TSK: 0
I20260619 13:01:16.259164 10218 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:01:16.520905 10209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:16.521174 10209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:16.521355 10209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:16.528848 10209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:16.528952 10209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:16.529110 10209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:01:16.539578 10209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.65.keytab
--principal=oryx/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:46001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=42607
--tserver_master_addrs=127.1.87.126:37527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:16.540964 10209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:16.542243 10209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:16.555747 10236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:16.555789 10234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:16.556115 10233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:16.557250 10209 server_base.cc:1034] running on GCE node
I20260619 13:01:16.557850 10209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:16.558856 10209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:16.560076 10209 hybrid_clock.cc:648] HybridClock initialized: now 1781874076560019 us; error 51 us; skew 500 ppm
I20260619 13:01:16.560393 10209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:16 dist-test-slave-tpsm krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874076, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:16.563604 10209 init.cc:377] Logged in from keytab as oryx/127.1.87.65@KRBTEST.COM (short username oryx)
I20260619 13:01:16.565145 10209 webserver.cc:460] Webserver started at http://127.1.87.65:42607/ using document root <none> and password file <none>
I20260619 13:01:16.565819 10209 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:16.565970 10209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:16.571305 10209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:01:16.574167 10243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:16.575639 10209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:01:16.575824 10209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "c24fab2e749542f5bd65d71d2d282b14"
format_stamp: "Formatted at 2026-06-19 13:01:13 on dist-test-slave-tpsm"
server_key: "c1068be60a2874405ebfe7a9f3e58fb1"
server_key_iv: "f7a1a0baaf45ea80ae51a544826e3f1b"
server_key_version: "encryptionkey@0"
I20260619 13:01:16.576805 10209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:16.614013 10209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:16.618422 10209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:16.618726 10209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:16.619748 10209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:16.621142 10209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:16.621270 10209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:16.621397 10209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:16.621490 10209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:16.654651 10209 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:46001
I20260619 13:01:16.654747 10355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:46001 every 8 connection(s)
I20260619 13:01:16.656173 10209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:01:16.660277 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 10209
I20260619 13:01:16.661105 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:41863
--local_ip_for_outbound_sockets=127.1.87.66
--tserver_master_addrs=127.1.87.126:37527
--webserver_port=35515
--webserver_interface=127.1.87.66
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.87.66
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:01:16 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:16 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:16.677366 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:16.659209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:59429)
0619 13:01:16.660892 (+ 1683us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:16.660940 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:16.663636 (+ 2696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:16.665033 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:16.665045 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:16.665829 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:16.666565 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:16.666590 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.668504 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.668508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:16.669693 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:16.669712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.671449 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.671458 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:16.671779 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:16.672476 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:16.672498 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:16.676408 (+ 3910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1342,"thread_start_us":362,"threads_started":1}
W20260619 13:01:16.678079 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:16.659158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:59429 (local address 127.1.87.126:37527)
0619 13:01:16.660028 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:16.660044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:16.661093 (+ 1049us) server_negotiation.cc:407] Connection header received
0619 13:01:16.664204 (+ 3111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:16.664217 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:16.664540 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:16.664826 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:16.666761 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.668332 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.669847 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.671008 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.671209 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:16.676859 (+ 5650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:59429: BlockingRecv error: recv got EOF from 127.1.87.65:59429 (error 108)
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":271,"threads_started":1}
W20260619 13:01:16.678570 10356 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:37527 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.87.126:37527: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Jun 19 13:01:16 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:16 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:16.689090 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:16.679326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:47183)
0619 13:01:16.679759 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:16.679775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:16.679862 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:16.680239 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:16.680242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:16.680638 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:16.680791 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:16.680801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.682519 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.682523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:16.683710 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:16.683724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.684785 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.684790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:16.685013 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:16.685717 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:16.685741 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:16.688804 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322}
W20260619 13:01:16.689180 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:16.679593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:47183 (local address 127.1.87.126:37527)
0619 13:01:16.679799 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:16.679807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:16.679833 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:01:16.679978 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:16.679986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:16.680075 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:16.680199 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:16.680936 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.682353 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.683855 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.684626 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.684777 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:16.688985 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:47183: BlockingRecv error: recv got EOF from 127.1.87.65:47183 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 13:01:16 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:16 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:16.697834 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:16.689821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:53123)
0619 13:01:16.689997 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:16.690016 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:16.690118 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:16.690487 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:16.690491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:16.690774 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:16.690915 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:16.690922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.692334 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.692338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:16.693257 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:16.693268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.693966 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.693972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:16.694165 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:16.694744 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:16.694760 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:16.697508 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 13:01:16.698004 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:16.689981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:53123 (local address 127.1.87.126:37527)
0619 13:01:16.690162 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:16.690168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:16.690185 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:01:16.690235 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:16.690241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:16.690323 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:16.690419 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:16.691047 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.692183 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.693404 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:16.693822 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:16.693956 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:16.697708 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:53123: BlockingRecv error: recv got EOF from 127.1.87.65:53123 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 13:01:16.698361 10356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:01:16.995600 10360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:16.995923 10360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:16.996105 10360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:17.003448 10360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:17.003597 10360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:17.003768 10360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:01:17.013774 10360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.66.keytab
--principal=oryx/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:41863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=35515
--tserver_master_addrs=127.1.87.126:37527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:17.015161 10360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:17.016507 10360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:17.029131 10368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:17.029299 10365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:17.029217 10366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:17.030361 10360 server_base.cc:1034] running on GCE node
I20260619 13:01:17.030936 10360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:17.031909 10360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:17.033179 10360 hybrid_clock.cc:648] HybridClock initialized: now 1781874077033128 us; error 61 us; skew 500 ppm
I20260619 13:01:17.033536 10360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874077, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:17.036828 10360 init.cc:377] Logged in from keytab as oryx/127.1.87.66@KRBTEST.COM (short username oryx)
I20260619 13:01:17.038323 10360 webserver.cc:460] Webserver started at http://127.1.87.66:35515/ using document root <none> and password file <none>
I20260619 13:01:17.039008 10360 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:17.039183 10360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:17.044466 10360 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:01:17.047736 10375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:17.049364 10360 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:01:17.049557 10360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "a26c5d8747b3463890c4b4e50739d91f"
format_stamp: "Formatted at 2026-06-19 13:01:14 on dist-test-slave-tpsm"
server_key: "142d2a583d250c36836e212366abc89d"
server_key_iv: "5749ccdd84a3f65a622f3b55020b25a1"
server_key_version: "encryptionkey@0"
I20260619 13:01:17.050540 10360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:17.070874 10360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:17.075855 10360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:17.076165 10360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:17.077170 10360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:17.078635 10360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:17.078764 10360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:17.078914 10360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:17.079007 10360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:17.113727 10360 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:41863
I20260619 13:01:17.113814 10487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:41863 every 8 connection(s)
I20260619 13:01:17.115141 10360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:01:17.116344 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 10360
I20260619 13:01:17.117030 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:35673
--local_ip_for_outbound_sockets=127.1.87.67
--tserver_master_addrs=127.1.87.126:37527
--webserver_port=34817
--webserver_interface=127.1.87.67
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.87.67
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:01:17.132683 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:17.118025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:49681 (local address 127.1.87.126:37527)
0619 13:01:17.118252 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:17.118261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:17.119428 (+ 1167us) server_negotiation.cc:407] Connection header received
0619 13:01:17.121405 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:17.121414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:17.121508 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:17.121633 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:17.123245 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.124454 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.126058 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.126899 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.127056 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.132239 (+ 5183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:49681: BlockingRecv error: recv got EOF from 127.1.87.66:49681 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:01:17.133198 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:17.118438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:49681)
0619 13:01:17.119246 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:17.119290 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:17.121066 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:17.121882 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:17.121890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:17.122483 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:17.123030 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.123047 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.124654 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.124658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.125888 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.125901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.127134 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.127139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.127423 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.128110 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:17.128137 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:17.131960 (+ 3823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":361,"threads_started":1}
W20260619 13:01:17.134078 10488 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:37527 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.87.126:37527: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:17.147302 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:17.134827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:46547)
0619 13:01:17.135176 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:17.135201 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:17.135373 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:17.137006 (+ 1633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:17.137014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:17.137521 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:17.137760 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.137777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.138802 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.138810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.140596 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.140613 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.141366 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.141372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.141561 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.142259 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:17.142286 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:17.146912 (+ 4626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182}
W20260619 13:01:17.147567 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:17.135070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:46547 (local address 127.1.87.126:37527)
0619 13:01:17.135209 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:17.135214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:17.135470 (+ 256us) server_negotiation.cc:407] Connection header received
0619 13:01:17.136770 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:17.136775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:17.136845 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:17.136934 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:17.137927 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.138653 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.140771 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.141186 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.141314 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.147117 (+ 5803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:46547: BlockingRecv error: recv got EOF from 127.1.87.66:46547 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:17.158567 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:17.148386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:44663)
0619 13:01:17.148586 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:17.148608 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:17.148736 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:17.149497 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:17.149502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:17.149803 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:17.149976 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.149984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.151613 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.151618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.152514 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.152524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.153542 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.153549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.153844 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.154712 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:17.154732 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:17.158326 (+ 3594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 13:01:17.158933 10488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:01:17.159128 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:17.148707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:44663 (local address 127.1.87.126:37527)
0619 13:01:17.148992 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:17.149000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:17.149029 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:01:17.149119 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:17.149125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:17.149234 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:17.149466 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:17.150179 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.151449 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.152654 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.153317 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.153443 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.158604 (+ 5161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:44663: BlockingRecv error: recv got EOF from 127.1.87.66:44663 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 13:01:17.462426 10491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:17.462697 10491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:17.462791 10491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:17.470057 10491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:17.470175 10491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:17.470288 10491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:01:17.479866 10491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/krb5kdc/oryx_127.1.87.67.keytab
--principal=oryx/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:35673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=34817
--tserver_master_addrs=127.1.87.126:37527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:17.481189 10491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:17.482447 10491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:17.496735 10496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:17.497080 10497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:17.496877 10499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:17.498023 10491 server_base.cc:1034] running on GCE node
I20260619 13:01:17.498664 10491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:17.499720 10491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:17.500959 10491 hybrid_clock.cc:648] HybridClock initialized: now 1781874077500913 us; error 62 us; skew 500 ppm
I20260619 13:01:17.501294 10491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874077, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:17.508438 10491 init.cc:377] Logged in from keytab as oryx/127.1.87.67@KRBTEST.COM (short username oryx)
I20260619 13:01:17.510025 10491 webserver.cc:460] Webserver started at http://127.1.87.67:34817/ using document root <none> and password file <none>
I20260619 13:01:17.510825 10491 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:17.510995 10491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:17.516417 10491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:01:17.519119 10506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:17.520574 10491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:01:17.520761 10491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "737ae9740cf442a186be113ddedc2d16"
format_stamp: "Formatted at 2026-06-19 13:01:15 on dist-test-slave-tpsm"
server_key: "b7b3c6d4f9ab08c27120db7dd000bca5"
server_key_iv: "4aa8478c4f1a14e4708cb122d9a547db"
server_key_version: "encryptionkey@0"
I20260619 13:01:17.521793 10491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:17.540872 10491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:17.545346 10491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:17.545661 10491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:17.546808 10491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:17.548192 10491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:17.548316 10491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:17.548447 10491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:17.548535 10491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:17.582736 10491 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:35673
I20260619 13:01:17.582789 10618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:35673 every 8 connection(s)
I20260619 13:01:17.584218 10491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:01:17.591547 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 10491
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:01:17.602025 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:17.586815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:42055 (local address 127.1.87.126:37527)
0619 13:01:17.586995 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:17.587000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:17.588160 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 13:01:17.590072 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:17.590082 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:17.590182 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:17.590299 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:17.592049 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.593197 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.595579 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.596249 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.596374 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.601685 (+ 5311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:42055: BlockingRecv error: recv got EOF from 127.1.87.67:42055 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":64}
I20260619 13:01:17.602682 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:17.587194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:42055)
0619 13:01:17.588017 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:17.588052 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:17.589717 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:17.590523 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:17.590531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:17.591015 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:17.591861 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.591887 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.593360 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.593368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.595454 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.595465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.596476 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.596481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.596833 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.597519 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:17.597547 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:17.601492 (+ 3945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":428,"threads_started":1}
W20260619 13:01:17.603446 10619 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:37527 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.87.126:37527: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
I20260619 13:01:17.608444 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:17.592559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.1.87.126:37527)
0619 13:01:17.593291 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:17.593296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:17.593314 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:01:17.593366 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:17.593369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:17.593418 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:17.593491 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:17.595291 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.596442 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.598176 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.598552 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.598639 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.601393 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:17.601400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:17.601418 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:17.601456 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:17.605150 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:17.605842 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:17.605851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:17.605860 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:17.606063 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:17.606445 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:17.606452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:17.606455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:17.606895 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:17.607199 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:17.607739 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 13:01:17.608106 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":615,"thread_start_us":510,"threads_started":1}
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:17.613061 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:17.604084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:38845)
0619 13:01:17.604343 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:17.604361 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:17.604463 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:17.604771 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:17.604778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:17.605183 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:17.605368 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.605376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.606331 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.606336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.607210 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.607220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.607951 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.607957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.608158 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.609700 (+ 1542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:17.609716 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:17.612782 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145}
W20260619 13:01:17.613212 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:17.604213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:38845 (local address 127.1.87.126:37527)
0619 13:01:17.604380 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:17.604385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:17.604404 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:01:17.604573 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:17.604576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:17.604634 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:17.604724 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:17.605506 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.606202 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.607374 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.607808 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.607941 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.612972 (+ 5031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:38845: BlockingRecv error: recv got EOF from 127.1.87.67:38845 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:17.621219 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:17.613749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:41041)
0619 13:01:17.613897 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:17.613933 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:17.614033 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:17.614324 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:17.614329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:17.614626 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:17.614775 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.614783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.615850 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.615857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.616852 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.616863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.617513 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.617519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.617718 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.618235 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:17.618251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:17.620944 (+ 2693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:01:17.621516 10619 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:01:17.621536 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:17.613821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:41041 (local address 127.1.87.126:37527)
0619 13:01:17.614045 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:17.614051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:17.614068 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:01:17.614122 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:17.614127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:17.614204 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:17.614303 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:17.614906 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.615724 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.616987 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.617373 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.617494 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.621228 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:41041: BlockingRecv error: recv got EOF from 127.1.87.67:41041 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:17 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:17.707368 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:17.698958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:38981)
0619 13:01:17.699516 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:17.699532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:17.699649 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:17.699981 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:17.699985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:17.700451 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:17.700639 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.700647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.701758 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.701762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.702739 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:17.702751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.703514 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.703519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:17.703644 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.704269 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:17.704286 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:17.707013 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":274,"threads_started":1}
W20260619 13:01:17.707439 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:17.699018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:38981 (local address 127.1.87.126:37527)
0619 13:01:17.699166 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:17.699170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:17.699634 (+ 464us) server_negotiation.cc:407] Connection header received
0619 13:01:17.699747 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:17.699751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:17.699820 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:17.699908 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:17.700779 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.701591 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.702862 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:17.703277 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:17.703468 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:17.707202 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:38981: BlockingRecv error: recv got EOF from 127.1.87.65:38981 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 13:01:18 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:18 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:18.168423 10625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:18.159573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:36899)
0619 13:01:18.160082 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:18.160097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:18.160262 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:18.160596 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:18.160601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:18.161064 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:18.161242 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:18.161251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.162246 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.162252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:18.163191 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:18.163201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.164096 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.164102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:18.164260 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:18.164880 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:18.164898 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:18.168083 (+ 3185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":270,"threads_started":1}
W20260619 13:01:18.168459 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:18.159654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:36899 (local address 127.1.87.126:37527)
0619 13:01:18.159916 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:18.159926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:18.160193 (+ 267us) server_negotiation.cc:407] Connection header received
0619 13:01:18.160389 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:18.160394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:18.160461 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:18.160576 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:18.161385 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.162109 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.163373 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.163902 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.164027 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:18.168270 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:36899: BlockingRecv error: recv got EOF from 127.1.87.66:36899 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 19 13:01:18 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:18 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:18.631718 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:18.622207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:35177)
0619 13:01:18.622742 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:18.622759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:18.622892 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:18.623207 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:18.623210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:18.623762 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:18.624034 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:18.624047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.625589 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.625597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:18.626773 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:18.626786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.627511 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.627516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:18.627640 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:18.628188 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:18.628205 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:18.631489 (+ 3284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
W20260619 13:01:18.631943 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:18.622289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:35177 (local address 127.1.87.126:37527)
0619 13:01:18.622544 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:18.622561 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:18.622853 (+ 292us) server_negotiation.cc:407] Connection header received
0619 13:01:18.623007 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:18.623011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:18.623075 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:18.623158 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:18.624174 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.625323 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.626907 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.627294 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.627464 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:18.631674 (+ 4210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:35177: BlockingRecv error: recv got EOF from 127.1.87.67:35177 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 13:01:18 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:18 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:18.716540 10627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:18.708439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:51543)
0619 13:01:18.708948 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:18.708967 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:18.709107 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:18.709482 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:18.709487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:18.709879 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:18.710017 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:18.710025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.711028 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.711035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:18.711992 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:18.712003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.712674 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.712679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:18.712844 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:18.713378 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:18.713394 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:18.716194 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":266,"threads_started":1}
W20260619 13:01:18.716665 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:18.708486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:51543 (local address 127.1.87.126:37527)
0619 13:01:18.708700 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:18.708709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:18.709034 (+ 325us) server_negotiation.cc:407] Connection header received
0619 13:01:18.709240 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:18.709260 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:18.709318 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:18.709422 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:18.710157 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.710901 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.712131 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:18.712526 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:18.712660 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:18.716414 (+ 3754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:51543: BlockingRecv error: recv got EOF from 127.1.87.65:51543 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 13:01:19 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:19 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:19.178057 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:19.169390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:43785)
0619 13:01:19.169904 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:19.169918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:19.170020 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:19.170342 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:19.170346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:19.170712 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:19.170839 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:19.170846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.172238 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.172242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:19.173117 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:19.173127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.174068 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.174072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:19.174186 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:19.174786 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:19.174803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:19.177709 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":303,"threads_started":1}
W20260619 13:01:19.178298 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:19.169451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:43785 (local address 127.1.87.126:37527)
0619 13:01:19.169626 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:19.169632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:19.169988 (+ 356us) server_negotiation.cc:407] Connection header received
0619 13:01:19.170122 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:19.170127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:19.170217 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:19.170318 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:19.170967 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.172122 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.173243 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.173921 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.174034 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:19.177949 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:43785: BlockingRecv error: recv got EOF from 127.1.87.66:43785 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 13:01:19 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:19 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:19.640703 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:19.632615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:58623)
0619 13:01:19.633298 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:19.633317 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:19.633459 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:19.633822 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:19.633826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:19.634328 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:19.634466 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:19.634475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.635492 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.635499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:19.636388 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:19.636399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.637101 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.637107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:19.637269 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:19.637788 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:19.637804 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:19.640349 (+ 2545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":456,"threads_started":1}
W20260619 13:01:19.640792 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:19.632777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:58623 (local address 127.1.87.126:37527)
0619 13:01:19.632958 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:19.632966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:19.633420 (+ 454us) server_negotiation.cc:407] Connection header received
0619 13:01:19.633588 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:19.633594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:19.633673 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:19.633777 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:19.634612 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.635357 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.636530 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.636950 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.637088 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:19.640531 (+ 3443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:58623: BlockingRecv error: recv got EOF from 127.1.87.67:58623 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 13:01:19 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:19 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:19.725630 10630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:19.717528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:44687)
0619 13:01:19.718003 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:19.718018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:19.718173 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:19.718507 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:19.718513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:19.718872 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:19.719033 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:19.719041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.720117 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.720125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:19.721073 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:19.721085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.721789 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.721794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:19.721971 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:19.722606 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:19.722623 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:19.725341 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":251,"threads_started":1}
W20260619 13:01:19.725821 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:19.717572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:44687 (local address 127.1.87.126:37527)
0619 13:01:19.717806 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:19.717814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:19.718105 (+ 291us) server_negotiation.cc:407] Connection header received
0619 13:01:19.718295 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:19.718300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:19.718372 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:19.718473 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:19.719180 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.719972 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.721207 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:19.721603 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:19.721778 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:19.725543 (+ 3765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:44687: BlockingRecv error: recv got EOF from 127.1.87.65:44687 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 13:01:20 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:20 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:20.187481 10631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:20.179068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:39411)
0619 13:01:20.179624 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:20.179658 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:20.180048 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:20.180392 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:20.180396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:20.180810 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:20.181025 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:20.181034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.182028 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.182034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:20.182921 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:20.182931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.183644 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.183649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:20.183818 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:20.184377 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:20.184393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:20.187094 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":350,"threads_started":1}
W20260619 13:01:20.187556 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:20.179173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:39411 (local address 127.1.87.126:37527)
0619 13:01:20.179404 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:20.179412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:20.179749 (+ 337us) server_negotiation.cc:407] Connection header received
0619 13:01:20.180184 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:20.180190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:20.180261 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:20.180366 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:20.181159 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.181894 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.183059 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.183506 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.183634 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:20.187305 (+ 3671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:39411: BlockingRecv error: recv got EOF from 127.1.87.66:39411 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 13:01:20 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:20 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:20.650333 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:20.641714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:53623)
0619 13:01:20.642235 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:20.642257 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:20.642504 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:20.642878 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:20.642883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:20.643258 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:20.643486 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:20.643495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.644660 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.644667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:20.645635 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:20.645646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.646347 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.646353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:20.646531 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:20.647175 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:20.647195 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:20.649974 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":287,"threads_started":1}
W20260619 13:01:20.650424 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:20.641799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:53623 (local address 127.1.87.126:37527)
0619 13:01:20.642027 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:20.642034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:20.642358 (+ 324us) server_negotiation.cc:407] Connection header received
0619 13:01:20.642642 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:20.642648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:20.642745 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:20.642848 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:20.643627 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.644534 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.645784 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.646200 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.646332 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:20.650185 (+ 3853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:53623: BlockingRecv error: recv got EOF from 127.1.87.67:53623 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 13:01:20 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:20 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:20.734628 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:20.726596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:56273)
0619 13:01:20.727096 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:20.727112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:20.727228 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:20.727569 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:20.727573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:20.727928 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:20.728082 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:20.728090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.729156 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.729161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:20.730169 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:20.730187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.730859 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.730864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:20.731005 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:20.731706 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:20.731730 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:20.734398 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":278,"threads_started":1}
W20260619 13:01:20.734930 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:20.726650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:56273 (local address 127.1.87.126:37527)
0619 13:01:20.726831 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:20.726835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:20.727187 (+ 352us) server_negotiation.cc:407] Connection header received
0619 13:01:20.727369 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:20.727375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:20.727445 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:20.727525 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:20.728220 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.728938 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.730304 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:20.730715 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:20.730809 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:20.734627 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:56273: BlockingRecv error: recv got EOF from 127.1.87.65:56273 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 13:01:21 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:21 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:21.196509 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:21.188526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:36321)
0619 13:01:21.189030 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:21.189057 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:21.189161 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:21.189539 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:21.189544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:21.189960 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:21.190092 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:21.190100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.191034 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.191042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:21.192006 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:21.192016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.192701 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.192716 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:21.192909 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:21.193457 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:21.193474 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:21.196187 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":280,"threads_started":1}
W20260619 13:01:21.196599 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:21.188596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:36321 (local address 127.1.87.126:37527)
0619 13:01:21.188774 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:21.188780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:21.189123 (+ 343us) server_negotiation.cc:407] Connection header received
0619 13:01:21.189324 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:21.189327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:21.189388 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:21.189496 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:21.190248 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.190915 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.192146 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.192566 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.192692 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:21.196381 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:36321: BlockingRecv error: recv got EOF from 127.1.87.66:36321 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 13:01:21 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:21 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:21.660557 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:21.651379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:48751)
0619 13:01:21.652014 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:21.652029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:21.652141 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:21.652491 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:21.652495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:21.652944 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:21.653121 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:21.653130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.654469 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.654474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:21.656009 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:21.656023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.656734 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.656740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:21.656919 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:21.657551 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:21.657573 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:21.660345 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":290,"threads_started":1}
W20260619 13:01:21.660808 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:21.651806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:48751 (local address 127.1.87.126:37527)
0619 13:01:21.651949 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:21.651954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:21.652107 (+ 153us) server_negotiation.cc:407] Connection header received
0619 13:01:21.652241 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:21.652245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:21.652320 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:21.652404 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:21.653263 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.654327 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.656154 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.656587 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.656687 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:21.660539 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:48751: BlockingRecv error: recv got EOF from 127.1.87.67:48751 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 13:01:21 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:21 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:21.744441 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:21.735769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:41353)
0619 13:01:21.736293 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:21.736309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:21.736465 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:21.736830 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:21.736835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:21.737259 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:21.737407 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:21.737416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.738672 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.738680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:21.739651 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:21.739662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.740354 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.740359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:21.740525 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:21.741079 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:21.741096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:21.744192 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":274,"threads_started":1}
W20260619 13:01:21.744621 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:21.735845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:41353 (local address 127.1.87.126:37527)
0619 13:01:21.736123 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:21.736130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:21.736395 (+ 265us) server_negotiation.cc:407] Connection header received
0619 13:01:21.736617 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:21.736622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:21.736691 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:21.736795 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:21.737557 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.738533 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.739785 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:21.740214 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:21.740342 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:21.744407 (+ 4065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:41353: BlockingRecv error: recv got EOF from 127.1.87.65:41353 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 19 13:01:22 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:22 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:22.205611 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:22.197432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:56249)
0619 13:01:22.197970 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:22.197986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:22.198131 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:22.198459 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:22.198464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:22.198885 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:22.199033 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:22.199041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.200080 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.200088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:22.200988 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:22.200998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.201693 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.201699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:22.201869 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:22.202432 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:22.202448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:22.205375 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":237,"threads_started":1}
W20260619 13:01:22.205896 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:22.197658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:56249 (local address 127.1.87.126:37527)
0619 13:01:22.197892 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:22.197898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:22.198066 (+ 168us) server_negotiation.cc:407] Connection header received
0619 13:01:22.198256 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:22.198263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:22.198331 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:22.198432 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:22.199182 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.199954 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.201130 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.201554 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.201682 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:22.205600 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:56249: BlockingRecv error: recv got EOF from 127.1.87.66:56249 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 13:01:22 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:22 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:22.673887 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:22.661526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:37305)
0619 13:01:22.662048 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:22.662064 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:22.662176 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:22.662578 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:22.662582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:22.663033 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:22.663204 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:22.663214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.664264 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.664268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:22.665176 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:22.665187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.665897 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.665902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:22.666050 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:22.670569 (+ 4519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:22.670594 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:22.673653 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":254,"threads_started":1}
W20260619 13:01:22.674084 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:22.661629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:37305 (local address 127.1.87.126:37527)
0619 13:01:22.661795 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:22.661799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:22.662144 (+ 345us) server_negotiation.cc:407] Connection header received
0619 13:01:22.662317 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:22.662321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:22.662393 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:22.662476 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:22.663398 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.664102 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.665319 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.665743 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.665842 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:22.673829 (+ 7987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:37305: BlockingRecv error: recv got EOF from 127.1.87.67:37305 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 13:01:22 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:22 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:22.753750 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:22.745554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:53189)
0619 13:01:22.746094 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:22.746110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:22.746221 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:22.746515 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:22.746518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:22.746877 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:22.747017 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:22.747025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.748039 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.748044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:22.748933 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:22.748944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.749665 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.749670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:22.749795 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:22.750467 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:22.750484 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:22.753443 (+ 2959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":317,"threads_started":1}
W20260619 13:01:22.753916 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:22.745660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:53189 (local address 127.1.87.126:37527)
0619 13:01:22.745859 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:22.745866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:22.746205 (+ 339us) server_negotiation.cc:407] Connection header received
0619 13:01:22.746317 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:22.746320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:22.746381 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:22.746462 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:22.747136 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.747890 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.749075 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:22.749504 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:22.749602 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:22.753646 (+ 4044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:53189: BlockingRecv error: recv got EOF from 127.1.87.65:53189 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 13:01:23 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:23 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:23.215564 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:23.206606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:48321)
0619 13:01:23.207112 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:23.207129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:23.207244 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:23.207549 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:23.207552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:23.207925 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:23.208070 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:23.208078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.209074 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.209078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:23.209968 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:23.209978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.211167 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.211175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:23.211384 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:23.212171 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:23.212195 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:23.215270 (+ 3075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":305,"threads_started":1}
W20260619 13:01:23.215879 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:23.206740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:48321 (local address 127.1.87.126:37527)
0619 13:01:23.207013 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:23.207020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:23.207210 (+ 190us) server_negotiation.cc:407] Connection header received
0619 13:01:23.207366 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:23.207370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:23.207435 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:23.207527 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:23.208208 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.208927 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.210141 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.210943 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.211102 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:23.215596 (+ 4494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:48321: BlockingRecv error: recv got EOF from 127.1.87.66:48321 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 13:01:23 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:23 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:23.683398 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:23.674914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:55307)
0619 13:01:23.675513 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:23.675530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:23.675640 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:23.675946 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:23.675949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:23.676326 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:23.676470 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:23.676478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.677495 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.677498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:23.678353 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:23.678363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.679142 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.679146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:23.679275 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:23.679905 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:23.679922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:23.683088 (+ 3166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":328,"threads_started":1}
W20260619 13:01:23.683652 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:23.675035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:55307 (local address 127.1.87.126:37527)
0619 13:01:23.675221 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:23.675225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:23.675621 (+ 396us) server_negotiation.cc:407] Connection header received
0619 13:01:23.675742 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:23.675746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:23.675819 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:23.675903 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:23.676626 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.677377 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.678500 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.678995 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.679110 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:23.683372 (+ 4262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:55307: BlockingRecv error: recv got EOF from 127.1.87.67:55307 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 13:01:23 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:23 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:23.765144 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:23.755274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:50945)
0619 13:01:23.755975 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:23.756017 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:23.756164 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:23.757027 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:23.757033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:23.757570 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:23.757827 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:23.757842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.759292 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.759298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:23.760365 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:23.760378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.761030 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.761034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:23.761202 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:23.761823 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:23.761841 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:23.764910 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":301,"threads_started":1}
W20260619 13:01:23.765540 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:23.755406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:50945 (local address 127.1.87.126:37527)
0619 13:01:23.755578 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:23.755584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:23.756090 (+ 506us) server_negotiation.cc:407] Connection header received
0619 13:01:23.756353 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:23.756358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:23.756667 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:23.756754 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:23.757971 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.759099 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.760493 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:23.760899 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:23.760992 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:23.765148 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:50945: BlockingRecv error: recv got EOF from 127.1.87.65:50945 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 13:01:24 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:24 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:24.224223 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:24.216358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:55343)
0619 13:01:24.216881 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:24.216897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:24.217001 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:24.217307 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:24.217310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:24.217668 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:24.217819 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:24.217826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.218777 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.218780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:24.219732 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:24.219743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.220320 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.220324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:24.220428 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:24.221135 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:24.221158 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:24.223974 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":261,"threads_started":1}
W20260619 13:01:24.224375 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:24.216452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:55343 (local address 127.1.87.126:37527)
0619 13:01:24.216631 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:24.216635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:24.216964 (+ 329us) server_negotiation.cc:407] Connection header received
0619 13:01:24.217092 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:24.217096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:24.217174 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:24.217256 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:24.217932 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.218668 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.219843 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.220199 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.220288 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:24.224152 (+ 3864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:55343: BlockingRecv error: recv got EOF from 127.1.87.66:55343 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 13:01:24 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:24 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:24.692677 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:24.684531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:50601)
0619 13:01:24.685146 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:24.685163 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:24.685308 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:24.685684 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:24.685690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:24.686084 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:24.686243 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:24.686251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.687229 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.687237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:24.688175 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:24.688185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.688910 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.688917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:24.689076 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:24.689630 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:24.689645 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:24.692308 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":289,"threads_started":1}
W20260619 13:01:24.692706 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:24.684596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:50601 (local address 127.1.87.126:37527)
0619 13:01:24.684819 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:24.684828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:24.685248 (+ 420us) server_negotiation.cc:407] Connection header received
0619 13:01:24.685444 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:24.685450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:24.685531 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:24.685633 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:24.686392 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.687091 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.688314 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.688765 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.688898 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:24.692496 (+ 3598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:50601: BlockingRecv error: recv got EOF from 127.1.87.67:50601 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 13:01:24 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:24 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:24.774318 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:24.765955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:39731)
0619 13:01:24.766508 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:24.766523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:24.766666 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:24.767010 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:24.767014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:24.767549 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:24.767711 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:24.767720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.768808 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.768814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:24.769721 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:24.769733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.770476 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.770482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:24.770661 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:24.771215 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:24.771231 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:24.773968 (+ 2737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":300,"threads_started":1}
W20260619 13:01:24.774397 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:24.766164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:39731 (local address 127.1.87.126:37527)
0619 13:01:24.766368 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:24.766376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:24.766600 (+ 224us) server_negotiation.cc:407] Connection header received
0619 13:01:24.766812 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:24.766821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:24.766879 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:24.766979 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:24.767845 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.768662 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.769861 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:24.770301 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:24.770464 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:24.774164 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:39731: BlockingRecv error: recv got EOF from 127.1.87.65:39731 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 13:01:25 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:25 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:25.233409 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:25.225120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:47571)
0619 13:01:25.225655 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:25.225671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:25.225820 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:25.226157 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:25.226161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:25.226607 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:25.226754 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:25.226762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.227961 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.227967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:25.228847 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:25.228857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.229686 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.229692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:25.229853 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:25.230422 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:25.230461 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:25.233180 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":304,"threads_started":1}
W20260619 13:01:25.233599 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:25.225204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:47571 (local address 127.1.87.126:37527)
0619 13:01:25.225428 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:25.225435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:25.225752 (+ 317us) server_negotiation.cc:407] Connection header received
0619 13:01:25.225955 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:25.225961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:25.226030 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:25.226140 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:25.226892 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.227822 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.228982 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.229505 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.229637 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:25.233357 (+ 3720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:47571: BlockingRecv error: recv got EOF from 127.1.87.66:47571 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 13:01:25 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:25 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:25.702149 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:25.693792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:50249)
0619 13:01:25.694310 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:25.694326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:25.694482 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:25.694842 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:25.694847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:25.695266 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:25.695486 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:25.695494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.696491 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.696499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:25.697381 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:25.697391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.698248 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.698255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:25.698434 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:25.699098 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:25.699116 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:25.701816 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":277,"threads_started":1}
W20260619 13:01:25.702278 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:25.693865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:50249 (local address 127.1.87.126:37527)
0619 13:01:25.694126 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:25.694134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:25.694407 (+ 273us) server_negotiation.cc:407] Connection header received
0619 13:01:25.694631 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:25.694637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:25.694711 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:25.694813 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:25.695639 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.696351 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.697523 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.698071 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.698214 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:25.702005 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:50249: BlockingRecv error: recv got EOF from 127.1.87.67:50249 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 13:01:25 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:25 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:25.783886 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:25.775410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:51765)
0619 13:01:25.775978 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:25.775994 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:25.776109 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:25.776449 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:25.776453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:25.776896 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:25.777063 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:25.777071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.778190 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.778197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:25.779230 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:25.779242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.779978 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.779983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:25.780158 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:25.780744 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:25.780761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:25.783623 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":321,"threads_started":1}
W20260619 13:01:25.784053 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:25.775453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:51765 (local address 127.1.87.126:37527)
0619 13:01:25.775608 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:25.775613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:25.776063 (+ 450us) server_negotiation.cc:407] Connection header received
0619 13:01:25.776204 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:25.776208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:25.776266 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:25.776353 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:25.777214 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.778057 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.779424 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:25.779828 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:25.779961 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:25.783836 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:51765: BlockingRecv error: recv got EOF from 127.1.87.65:51765 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 13:01:26 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:26 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:26.242951 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:26.234368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:53283)
0619 13:01:26.234870 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:26.234886 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:26.235047 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:26.235390 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:26.235394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:26.235787 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:26.235937 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:26.235944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.236933 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.236941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:26.238111 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:26.238122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.239053 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.239061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:26.239234 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:26.239855 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:26.239872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:26.242687 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":282,"threads_started":1}
W20260619 13:01:26.243054 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:26.234427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:53283 (local address 127.1.87.126:37527)
0619 13:01:26.234652 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:26.234660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:26.234973 (+ 313us) server_negotiation.cc:407] Connection header received
0619 13:01:26.235171 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:26.235178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:26.235242 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:26.235367 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:26.236082 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.236810 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.238280 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.238860 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.238991 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:26.242865 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:53283: BlockingRecv error: recv got EOF from 127.1.87.66:53283 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 13:01:26 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:26 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:26.710938 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:26.703143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:40745)
0619 13:01:26.703689 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:26.703705 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:26.703832 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:26.704132 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:26.704135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:26.704499 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:26.704657 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:26.704664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.705655 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.705669 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:26.706610 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:26.706621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.707259 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.707263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:26.707464 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:26.708052 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:26.708070 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:26.710698 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":318,"threads_started":1}
W20260619 13:01:26.711586 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:26.703159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:40745 (local address 127.1.87.126:37527)
0619 13:01:26.703394 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:26.703403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:26.703801 (+ 398us) server_negotiation.cc:407] Connection header received
0619 13:01:26.703934 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:26.703938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:26.703998 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:26.704075 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:26.704788 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.705522 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.706736 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.707138 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.707230 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:26.711180 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:40745: BlockingRecv error: recv got EOF from 127.1.87.67:40745 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 13:01:26 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:26 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:26.793015 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:26.784747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:60307)
0619 13:01:26.785483 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:26.785499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:26.785643 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:26.785959 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:26.785963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:26.786332 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:26.786468 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:26.786476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.787510 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.787516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:26.788425 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:26.788436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.789184 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.789189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:26.789367 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:26.789902 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:26.789917 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:26.792670 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":524,"threads_started":1}
W20260619 13:01:26.793134 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:26.784845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:60307 (local address 127.1.87.126:37527)
0619 13:01:26.785067 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:26.785077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:26.785587 (+ 510us) server_negotiation.cc:407] Connection header received
0619 13:01:26.785768 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:26.785774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:26.785836 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:26.785930 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:26.786611 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.787377 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.788567 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:26.789026 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:26.789156 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:26.792887 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:60307: BlockingRecv error: recv got EOF from 127.1.87.65:60307 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 13:01:27 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:27 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:27.251962 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:27.243969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:42497)
0619 13:01:27.244393 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:27.244409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:27.244513 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:27.244817 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:27.244821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:27.245210 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:27.245345 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:27.245352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.246332 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.246336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:27.247220 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:27.247230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.248266 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.248271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:27.248408 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:27.248941 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:27.248957 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:27.251664 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":253,"threads_started":1}
W20260619 13:01:27.252162 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:27.244091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:42497 (local address 127.1.87.126:37527)
0619 13:01:27.244330 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:27.244337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:27.244481 (+ 144us) server_negotiation.cc:407] Connection header received
0619 13:01:27.244603 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:27.244606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:27.244664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:27.244776 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:27.245460 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.246219 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.247547 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.248078 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.248243 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:27.251905 (+ 3662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:42497: BlockingRecv error: recv got EOF from 127.1.87.66:42497 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 19 13:01:27 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:27 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:27.725296 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:27.711895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:37595)
0619 13:01:27.712428 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:27.712443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:27.712547 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:27.712797 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:27.712800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:27.713141 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:27.713279 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:27.713287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.714366 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.714371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:27.715667 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:27.715685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.716507 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.716511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:27.716644 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:27.722294 (+ 5650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:27.722322 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:27.725065 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":317,"threads_started":1}
W20260619 13:01:27.725543 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:27.712039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:37595 (local address 127.1.87.126:37527)
0619 13:01:27.712221 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:27.712226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:27.712527 (+ 301us) server_negotiation.cc:407] Connection header received
0619 13:01:27.712639 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:27.712643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:27.712699 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:27.712783 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:27.713438 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.714190 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.715808 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.716348 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.716444 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:27.725268 (+ 8824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:37595: BlockingRecv error: recv got EOF from 127.1.87.67:37595 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 13:01:27 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:27 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:27.802659 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:27.793997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:39183)
0619 13:01:27.794533 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:27.794549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:27.794700 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:27.795033 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:27.795038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:27.795585 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:27.795785 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:27.795794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.796738 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.796745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:27.797631 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:27.797642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.798370 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.798376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:27.798541 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:27.799293 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:27.799343 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:27.802368 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":305,"threads_started":1}
W20260619 13:01:27.802845 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:27.794103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:39183 (local address 127.1.87.126:37527)
0619 13:01:27.794360 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:27.794370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:27.794629 (+ 259us) server_negotiation.cc:407] Connection header received
0619 13:01:27.794843 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:27.794847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:27.794904 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:27.795002 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:27.795915 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.796611 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.797767 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:27.798213 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:27.798337 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:27.802582 (+ 4245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:39183: BlockingRecv error: recv got EOF from 127.1.87.65:39183 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 13:01:28 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:28 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:28.260861 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:28.252956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:36831)
0619 13:01:28.253522 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:28.253543 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:28.253666 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:28.253972 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:28.253976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:28.254362 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:28.254607 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:28.254620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.255654 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.255658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:28.256525 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:28.256535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.257137 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.257141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:28.257246 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:28.257871 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:28.257888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:28.260623 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":286,"threads_started":1}
W20260619 13:01:28.261049 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:28.253005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:36831 (local address 127.1.87.126:37527)
0619 13:01:28.253173 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:28.253180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:28.253621 (+ 441us) server_negotiation.cc:407] Connection header received
0619 13:01:28.253761 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:28.253764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:28.253828 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:28.253910 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:28.254737 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.255532 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.256645 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.257018 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.257107 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:28.260811 (+ 3704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:36831: BlockingRecv error: recv got EOF from 127.1.87.66:36831 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 13:01:28 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:28 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:28.734469 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:28.726222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:59723)
0619 13:01:28.726741 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:28.726756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:28.726864 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:28.727153 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:28.727156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:28.727585 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:28.727750 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:28.727758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.729176 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.729183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:28.730287 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:28.730308 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.730929 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.730932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:28.731037 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:28.731625 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:28.731642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:28.734248 (+ 2606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":303,"threads_started":1}
W20260619 13:01:28.734647 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:28.726391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:59723 (local address 127.1.87.126:37527)
0619 13:01:28.726574 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:28.726578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:28.726824 (+ 246us) server_negotiation.cc:407] Connection header received
0619 13:01:28.726957 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:28.726961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:28.727019 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:28.727108 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:28.727881 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.729040 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.730429 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.730802 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.730900 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:28.734425 (+ 3525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:59723: BlockingRecv error: recv got EOF from 127.1.87.67:59723 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 13:01:28 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:28 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:28.811611 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:28.803682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:36873)
0619 13:01:28.804165 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:28.804181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:28.804292 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:28.804653 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:28.804657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:28.805101 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:28.805248 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:28.805256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.806292 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.806296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:28.807169 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:28.807179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.807861 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.807864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:28.807984 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:28.808523 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:28.808540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:28.811232 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":271,"threads_started":1}
W20260619 13:01:28.811863 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:28.803741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:36873 (local address 127.1.87.126:37527)
0619 13:01:28.803991 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:28.803998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:28.804259 (+ 261us) server_negotiation.cc:407] Connection header received
0619 13:01:28.804431 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:28.804435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:28.804496 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:28.804583 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:28.805401 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.806167 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.807299 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:28.807752 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:28.807841 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:28.811586 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:36873: BlockingRecv error: recv got EOF from 127.1.87.65:36873 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 13:01:29 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:29 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:29.269851 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:29.261713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:59645)
0619 13:01:29.262204 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:29.262222 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:29.262351 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:29.262630 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:29.262633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:29.263027 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:29.263184 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:29.263193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.264153 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.264158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:29.265220 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:29.265232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.266016 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.266022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:29.266227 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:29.266787 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:29.266805 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:29.269467 (+ 2662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":273,"threads_started":1}
W20260619 13:01:29.269922 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:29.261880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:59645 (local address 127.1.87.126:37527)
0619 13:01:29.262106 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:29.262112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:29.262306 (+ 194us) server_negotiation.cc:407] Connection header received
0619 13:01:29.262450 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:29.262454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:29.262511 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:29.262590 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:29.263303 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.264017 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.265363 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.265861 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.266018 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:29.269664 (+ 3646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:59645: BlockingRecv error: recv got EOF from 127.1.87.66:59645 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 13:01:29 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:29 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:29.743041 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:29.735282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:58791)
0619 13:01:29.735785 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:29.735801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:29.735915 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:29.736198 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:29.736202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:29.736589 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:29.736739 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:29.736747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.737750 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.737754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:29.738650 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:29.738662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.739281 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.739285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:29.739468 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:29.740026 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:29.740050 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:29.742703 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":259,"threads_started":1}
W20260619 13:01:29.743144 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:29.735403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:58791 (local address 127.1.87.126:37527)
0619 13:01:29.735621 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:29.735628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:29.735882 (+ 254us) server_negotiation.cc:407] Connection header received
0619 13:01:29.736009 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:29.736013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:29.736072 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:29.736149 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:29.736862 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.737596 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.738769 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.739161 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.739258 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:29.742902 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:58791: BlockingRecv error: recv got EOF from 127.1.87.67:58791 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 13:01:29 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:29 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:29.821089 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:29.812604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:41385)
0619 13:01:29.813156 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:29.813172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:29.813354 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:29.813691 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:29.813696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:29.814121 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:29.814283 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:29.814291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.815298 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.815305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:29.816573 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:29.816585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.817384 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.817391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:29.817563 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:29.818179 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:29.818197 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:29.820862 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":278,"threads_started":1}
W20260619 13:01:29.821290 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:29.812764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:41385 (local address 127.1.87.126:37527)
0619 13:01:29.813012 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:29.813020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:29.813262 (+ 242us) server_negotiation.cc:407] Connection header received
0619 13:01:29.813481 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:29.813487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:29.813557 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:29.813660 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:29.814426 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.815162 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.816722 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:29.817202 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:29.817343 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:29.821034 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:41385: BlockingRecv error: recv got EOF from 127.1.87.65:41385 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 13:01:30 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:30 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:30.278496 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:30.270782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:57905)
0619 13:01:30.271394 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:30.271410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:30.271530 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:30.271808 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:30.271811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:30.272174 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:30.272321 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:30.272329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.273286 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.273290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:30.274220 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:30.274231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.274893 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.274898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:30.275014 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:30.275625 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:30.275685 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:30.278231 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":348,"threads_started":1}
W20260619 13:01:30.278697 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:30.270896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:57905 (local address 127.1.87.126:37527)
0619 13:01:30.271222 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:30.271228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:30.271484 (+ 256us) server_negotiation.cc:407] Connection header received
0619 13:01:30.271624 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:30.271628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:30.271684 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:30.271771 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:30.272440 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.273171 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.274343 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.274734 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.274833 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:30.278445 (+ 3612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:57905: BlockingRecv error: recv got EOF from 127.1.87.66:57905 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 13:01:30 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:30 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:30.752380 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:30.744030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:47955)
0619 13:01:30.744512 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:30.744527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:30.744670 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:30.745004 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:30.745009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:30.745425 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:30.745573 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:30.745581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.746558 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.746566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:30.747498 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:30.747509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.748222 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.748230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:30.748382 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:30.749064 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:30.749082 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:30.752030 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":240,"threads_started":1}
W20260619 13:01:30.752434 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:30.744101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:47955 (local address 127.1.87.126:37527)
0619 13:01:30.744318 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:30.744331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:30.744602 (+ 271us) server_negotiation.cc:407] Connection header received
0619 13:01:30.744781 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:30.744787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:30.744856 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:30.744954 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:30.745715 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.746432 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.747633 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.748035 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.748158 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:30.752205 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:47955: BlockingRecv error: recv got EOF from 127.1.87.67:47955 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 13:01:30 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:30 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:30.829614 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:30.821951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:54693)
0619 13:01:30.822406 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:30.822421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:30.822527 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:30.822808 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:30.822811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:30.823144 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:30.823342 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:30.823350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.824311 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.824315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:30.825237 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:30.825249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.825981 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.825985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:30.826095 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:30.826749 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:30.826765 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:30.829392 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":265,"threads_started":1}
W20260619 13:01:30.829826 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:30.821991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:54693 (local address 127.1.87.126:37527)
0619 13:01:30.822368 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:30.822375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:30.822492 (+ 117us) server_negotiation.cc:407] Connection header received
0619 13:01:30.822623 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:30.822627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:30.822691 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:30.822779 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:30.823460 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.824193 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.825397 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:30.825858 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:30.825966 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:30.829559 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:54693: BlockingRecv error: recv got EOF from 127.1.87.65:54693 (error 108)
Metrics: {"server-negotiator.queue_time_us":185}
Jun 19 13:01:31 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:31 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:31.287942 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:31.279540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:54315)
0619 13:01:31.280054 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:31.280072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:31.280197 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:31.280530 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:31.280534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:31.280907 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:31.281143 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:31.281154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.282140 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.282144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:31.283095 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:31.283107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.283924 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.283932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:31.284106 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:31.284796 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:31.284818 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:31.287700 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":256,"threads_started":1}
W20260619 13:01:31.288049 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:31.279629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:54315 (local address 127.1.87.126:37527)
0619 13:01:31.279851 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:31.279858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:31.280153 (+ 295us) server_negotiation.cc:407] Connection header received
0619 13:01:31.280294 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:31.280298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:31.280357 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:31.280437 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:31.281293 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.282003 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.283238 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.283765 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.283879 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:31.287891 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:54315: BlockingRecv error: recv got EOF from 127.1.87.66:54315 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 13:01:31 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:31 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.67@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:31.762553 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:31.753481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.67:34831)
0619 13:01:31.754009 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:31.754031 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:31.754152 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:31.754529 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:31.754533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:31.754961 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:31.755134 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:31.755143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.756512 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.756520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:31.757637 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:31.757649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.758352 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.758358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:31.758534 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:31.759205 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:31.759224 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:31.762298 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":289,"threads_started":1}
W20260619 13:01:31.762876 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:31.753592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:34831 (local address 127.1.87.126:37527)
0619 13:01:31.753764 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:31.753768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:31.754104 (+ 336us) server_negotiation.cc:407] Connection header received
0619 13:01:31.754268 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:31.754272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:31.754337 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:31.754436 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:31.755279 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.756212 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.757781 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.758199 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.758347 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:31.762572 (+ 4225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.67:34831: BlockingRecv error: recv got EOF from 127.1.87.67:34831 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 13:01:31 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:31 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.65@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:31.839159 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:31.830837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.65:42809)
0619 13:01:31.831424 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:31.831446 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:31.831588 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:31.832377 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:31.832384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:31.832783 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:31.832984 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:31.832993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.833961 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.833965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:31.834877 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:31.834887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.835663 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.835678 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:31.835814 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:31.836381 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:31.836398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:31.838906 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":293,"threads_started":1}
W20260619 13:01:31.839280 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:31.830898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:42809 (local address 127.1.87.126:37527)
0619 13:01:31.831134 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:31.831143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:31.831535 (+ 392us) server_negotiation.cc:407] Connection header received
0619 13:01:31.831932 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:31.831938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:31.832048 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:31.832175 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:31.833126 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.833831 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.835006 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:31.835513 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:31.835609 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:31.839077 (+ 3468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.65:42809: BlockingRecv error: recv got EOF from 127.1.87.65:42809 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 13:01:32 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:01:32 dist-test-slave-tpsm krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.87.66@KRBTEST.COM for oryx/127.1.87.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:01:32.297494 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:32.289116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37527 (local address 127.1.87.66:44145)
0619 13:01:32.289627 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:32.289642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:32.289797 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:32.290118 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:32.290123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:32.290539 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:32.290715 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:32.290723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.291755 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.291762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:32.292654 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:32.292665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.293380 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.293387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:32.293554 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:32.294243 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:32.294266 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:32.297128 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.87.126:37527: Server oryx/127.1.87.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":294,"threads_started":1}
W20260619 13:01:32.297533 10358 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:32.289246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:44145 (local address 127.1.87.126:37527)
0619 13:01:32.289482 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:32.289489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:32.289726 (+ 237us) server_negotiation.cc:407] Connection header received
0619 13:01:32.289926 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:32.289936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:32.289995 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:32.290095 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:32.290853 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.291618 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.292791 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.293242 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.293368 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:32.297311 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.87.66:44145: BlockingRecv error: recv got EOF from 127.1.87.66:44145 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:01:32.595008 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10209
I20260619 13:01:32.606446 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:32.596145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.1.87.65:46001)
0619 13:01:32.596683 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:32.596698 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:32.596725 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:01:32.596792 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:32.596799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:32.597002 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:32.597159 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:32.598436 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.599520 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.600632 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.601140 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.601241 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:32.603531 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:32.603540 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:32.603553 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:32.603582 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:32.605989 (+ 2407us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.87.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 13:01:32.606235 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59824: Request ticket server kudu/127.1.87.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":284,"threads_started":1}
W20260619 13:01:32.606603 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10209: Not authorized: Client connection negotiation failed: client connection to 127.1.87.65:46001: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.87.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 13:01:32.606822 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 10209
I20260619 13:01:32.623397 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10360
I20260619 13:01:32.637238 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:32.624545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.1.87.66:41863)
0619 13:01:32.625139 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:32.625146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:32.625175 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:01:32.625239 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:32.625246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:32.625520 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:32.625698 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:32.627503 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.628821 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.630278 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.631078 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.631248 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:32.633947 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:32.633956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:32.633992 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:32.634028 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:32.636513 (+ 2485us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.87.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 13:01:32.636862 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53050: Request ticket server kudu/127.1.87.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":317,"threads_started":1}
W20260619 13:01:32.637256 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10360: Not authorized: Client connection negotiation failed: client connection to 127.1.87.66:41863: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.87.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 13:01:32.637326 10670 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53050: Request ticket server kudu/127.1.87.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 13:01:32.637413 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 10360
I20260619 13:01:32.653640 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10491
I20260619 13:01:32.665046 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:32.654629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.1.87.67:35673)
0619 13:01:32.655205 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:32.655213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:32.655238 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:01:32.655294 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:32.655300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:32.655571 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:32.655725 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:32.656955 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.657976 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.659093 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:32.659761 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:32.659860 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:32.662176 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:32.662184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:32.662196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:32.662236 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:32.664548 (+ 2312us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.87.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 13:01:32.664791 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49056: Request ticket server kudu/127.1.87.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":333,"threads_started":1}
W20260619 13:01:32.665145 10671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49056: Request ticket server kudu/127.1.87.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 13:01:32.665171 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10491: Not authorized: Client connection negotiation failed: client connection to 127.1.87.67:35673: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.87.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 13:01:32.665343 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 10491
I20260619 13:01:32.681398 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10140
I20260619 13:01:32.682600 10202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:32.797156 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 10140
2026-06-19T13:01:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21083 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T13:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:01:32Z Disabled control of system clock
I20260619 13:01:32.873399 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37409
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 13:01:33.195271 10676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:33.195592 10676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:33.195725 10676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:33.202939 10676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:01:33.203039 10676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:33.203122 10676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:33.203219 10676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:01:33.203294 10676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:01:33.212435 10676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:37409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:33.213739 10676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:33.215586 10676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:33.227185 10684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:33.227267 10682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:33.227231 10681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:33.227736 10676 server_base.cc:1034] running on GCE node
I20260619 13:01:33.228571 10676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:33.229979 10676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:33.231205 10676 hybrid_clock.cc:648] HybridClock initialized: now 1781874093231157 us; error 52 us; skew 500 ppm
I20260619 13:01:33.231566 10676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:01:33.237816 10676 webserver.cc:460] Webserver started at http://127.1.87.126:46181/ using document root <none> and password file <none>
I20260619 13:01:33.238500 10676 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:33.238624 10676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:33.238943 10676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:33.241796 10676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "e339afa3bad94c6d8d0075107811cd84"
format_stamp: "Formatted at 2026-06-19 13:01:33 on dist-test-slave-tpsm"
server_key: "544aa83e7c4220513cf67bf643bbe85e"
server_key_iv: "98da73f6495960c73100d5b00ade9eea"
server_key_version: "encryptionkey@0"
I20260619 13:01:33.242564 10676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "e339afa3bad94c6d8d0075107811cd84"
format_stamp: "Formatted at 2026-06-19 13:01:33 on dist-test-slave-tpsm"
server_key: "544aa83e7c4220513cf67bf643bbe85e"
server_key_iv: "98da73f6495960c73100d5b00ade9eea"
server_key_version: "encryptionkey@0"
I20260619 13:01:33.247249 10676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 13:01:33.250458 10690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:33.251683 10676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 13:01:33.251857 10676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "e339afa3bad94c6d8d0075107811cd84"
format_stamp: "Formatted at 2026-06-19 13:01:33 on dist-test-slave-tpsm"
server_key: "544aa83e7c4220513cf67bf643bbe85e"
server_key_iv: "98da73f6495960c73100d5b00ade9eea"
server_key_version: "encryptionkey@0"
I20260619 13:01:33.252063 10676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:33.271134 10676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:33.272002 10676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:33.272286 10676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:33.289947 10676 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:37409
I20260619 13:01:33.289968 10741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:37409 every 8 connection(s)
I20260619 13:01:33.291244 10676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:01:33.293509 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 10676
I20260619 13:01:33.293711 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:01:33.294366 1373 external_mini_cluster.cc:1428] Setting key 7e60821456680a7b16dc51dc6991c274
I20260619 13:01:33.294878 10742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:33.308204 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:33.296206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.1.87.126:37409)
0619 13:01:33.297131 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:33.297145 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:33.297184 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:01:33.299121 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:33.299141 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:33.299621 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:33.299978 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:33.300654 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:33.301904 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:33.303759 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:33.304416 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:33.304554 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:33.305821 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:33.305829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:33.305842 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:33.306407 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:33.306486 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:33.306509 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:33.306602 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:33.306842 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 13:01:33.307250 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":347,"threads_started":1}
I20260619 13:01:33.308787 10742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84: Bootstrap starting.
I20260619 13:01:33.311800 10742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:33.312740 10742 log.cc:824] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:33.315507 10742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84: No bootstrap required, opened a new log
I20260619 13:01:33.322342 10742 raft_consensus.cc:348] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e339afa3bad94c6d8d0075107811cd84" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37409 } }
I20260619 13:01:33.322705 10742 raft_consensus.cc:374] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:33.322813 10742 raft_consensus.cc:729] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e339afa3bad94c6d8d0075107811cd84, State: Initialized, Role: FOLLOWER
I20260619 13:01:33.323479 10742 consensus_queue.cc:260] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e339afa3bad94c6d8d0075107811cd84" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37409 } }
I20260619 13:01:33.323712 10742 raft_consensus.cc:388] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:01:33.323858 10742 raft_consensus.cc:482] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:01:33.324033 10742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:33.326159 10742 raft_consensus.cc:504] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e339afa3bad94c6d8d0075107811cd84" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37409 } }
I20260619 13:01:33.326659 10742 leader_election.cc:302] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e339afa3bad94c6d8d0075107811cd84; no voters:
I20260619 13:01:33.327214 10742 leader_election.cc:288] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:01:33.327404 10747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:33.328069 10747 raft_consensus.cc:686] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [term 1 LEADER]: Becoming Leader. State: Replica: e339afa3bad94c6d8d0075107811cd84, State: Running, Role: LEADER
I20260619 13:01:33.328575 10747 consensus_queue.cc:237] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e339afa3bad94c6d8d0075107811cd84" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37409 } }
I20260619 13:01:33.329224 10742 sys_catalog.cc:564] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:01:33.331221 10748 sys_catalog.cc:455] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e339afa3bad94c6d8d0075107811cd84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e339afa3bad94c6d8d0075107811cd84" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37409 } } }
I20260619 13:01:33.331647 10748 sys_catalog.cc:458] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:33.331650 10749 sys_catalog.cc:455] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e339afa3bad94c6d8d0075107811cd84. Latest consensus state: current_term: 1 leader_uuid: "e339afa3bad94c6d8d0075107811cd84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e339afa3bad94c6d8d0075107811cd84" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37409 } } }
I20260619 13:01:33.331940 10749 sys_catalog.cc:458] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:33.334677 10755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:01:33.339807 10755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:01:33.348433 10755 catalog_manager.cc:1347] Generated new cluster ID: 15ebf3e07456478b818060072b4efec2
I20260619 13:01:33.348558 10755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:01:33.369108 10755 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:01:33.370396 10755 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:01:33.379226 10755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e339afa3bad94c6d8d0075107811cd84: Generated new TSK 0
I20260619 13:01:33.380448 10755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:01:33.393329 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37409
--builtin_ntp_servers=127.1.87.84:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 13:01:33.721800 10766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:33.722122 10766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:33.722275 10766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:33.730074 10766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:33.730216 10766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:33.730383 10766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:01:33.739971 10766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:33.741297 10766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:33.743016 10766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:33.756098 10774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:33.756181 10771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:33.756124 10772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:33.757234 10766 server_base.cc:1034] running on GCE node
I20260619 13:01:33.757841 10766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:33.758780 10766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:33.760010 10766 hybrid_clock.cc:648] HybridClock initialized: now 1781874093759950 us; error 52 us; skew 500 ppm
I20260619 13:01:33.760403 10766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:01:33.762639 10766 webserver.cc:460] Webserver started at http://127.1.87.65:34651/ using document root <none> and password file <none>
I20260619 13:01:33.763396 10766 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:33.763525 10766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:33.763844 10766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:33.766595 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "7d3794a784fc4505b21e93c29f74b9c5"
format_stamp: "Formatted at 2026-06-19 13:01:33 on dist-test-slave-tpsm"
server_key: "e2f366d8ce085074749abcb4aea32b7b"
server_key_iv: "46dbfe656601fd96138e567a7a3bdd49"
server_key_version: "encryptionkey@0"
I20260619 13:01:33.767445 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "7d3794a784fc4505b21e93c29f74b9c5"
format_stamp: "Formatted at 2026-06-19 13:01:33 on dist-test-slave-tpsm"
server_key: "e2f366d8ce085074749abcb4aea32b7b"
server_key_iv: "46dbfe656601fd96138e567a7a3bdd49"
server_key_version: "encryptionkey@0"
I20260619 13:01:33.772421 10766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:01:33.776067 10780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:33.777557 10766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:01:33.777743 10766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "7d3794a784fc4505b21e93c29f74b9c5"
format_stamp: "Formatted at 2026-06-19 13:01:33 on dist-test-slave-tpsm"
server_key: "e2f366d8ce085074749abcb4aea32b7b"
server_key_iv: "46dbfe656601fd96138e567a7a3bdd49"
server_key_version: "encryptionkey@0"
I20260619 13:01:33.777966 10766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:33.803437 10766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:33.804257 10766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:33.804558 10766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:33.805434 10766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:33.806843 10766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:33.806957 10766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:33.807076 10766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:33.807183 10766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:33.841073 10766 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:43327
I20260619 13:01:33.841195 10892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:43327 every 8 connection(s)
I20260619 13:01:33.842312 10766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:01:33.845364 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 10766
I20260619 13:01:33.845549 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:01:33.846170 1373 external_mini_cluster.cc:1428] Setting key c8d94cf2e4227a5e5eb0969e84890151
I20260619 13:01:33.848932 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37409
--builtin_ntp_servers=127.1.87.84:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 13:01:33.855680 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:33.844701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:58797 (local address 127.1.87.126:37409)
0619 13:01:33.845147 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:33.845155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:33.846432 (+ 1277us) server_negotiation.cc:407] Connection header received
0619 13:01:33.848362 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:33.848368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:33.848446 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:33.848554 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:33.849678 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:33.850966 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:33.852513 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:33.853229 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:33.853370 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:33.854668 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:33.854676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:33.854684 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:33.854708 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:33.854758 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:33.854775 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:33.854861 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:33.855085 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 13:01:33.855198 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":196,"threads_started":1}
I20260619 13:01:33.856820 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:33.845271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37409 (local address 127.1.87.65:58797)
0619 13:01:33.846281 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:33.846323 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:33.848114 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:33.848734 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:33.848741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:33.848824 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:33.849419 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:33.849435 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:33.851125 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:33.851129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:33.852370 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:33.852381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:33.853463 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:33.853468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:33.853666 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:33.854359 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:01:33.854361 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:33.854399 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:01:33.854475 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:33.854875 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:33.854881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:33.854887 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:01:33.855072 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 13:01:33.855444 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":346,"threads_started":1}
I20260619 13:01:33.858244 10893 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37409
I20260619 13:01:33.858527 10893 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:33.859225 10893 heartbeater.cc:511] Master 127.1.87.126:37409 requested a full tablet report, sending...
I20260619 13:01:33.861150 10707 ts_manager.cc:194] Registered new tserver with Master: 7d3794a784fc4505b21e93c29f74b9c5 (127.1.87.65:43327)
I20260619 13:01:33.862581 10707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.65:58797
W20260619 13:01:34.195276 10897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:34.195642 10897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:34.195765 10897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:34.202996 10897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:34.203106 10897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:34.203270 10897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:01:34.212651 10897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:34.213922 10897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:34.215694 10897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:34.228233 10902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:34.228271 10905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:34.228302 10903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:34.229544 10897 server_base.cc:1034] running on GCE node
I20260619 13:01:34.230439 10897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:34.231573 10897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:34.232802 10897 hybrid_clock.cc:648] HybridClock initialized: now 1781874094232758 us; error 36 us; skew 500 ppm
I20260619 13:01:34.233139 10897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:01:34.235543 10897 webserver.cc:460] Webserver started at http://127.1.87.66:40051/ using document root <none> and password file <none>
I20260619 13:01:34.236220 10897 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:34.236330 10897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:34.236651 10897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:34.239437 10897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "50a46613c7f648159d9cbba0b387c9bf"
format_stamp: "Formatted at 2026-06-19 13:01:34 on dist-test-slave-tpsm"
server_key: "fc71d3cea4727b72a9a350eb43fd6f23"
server_key_iv: "95ed87d252f3105aac22d3618223898f"
server_key_version: "encryptionkey@0"
I20260619 13:01:34.240203 10897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "50a46613c7f648159d9cbba0b387c9bf"
format_stamp: "Formatted at 2026-06-19 13:01:34 on dist-test-slave-tpsm"
server_key: "fc71d3cea4727b72a9a350eb43fd6f23"
server_key_iv: "95ed87d252f3105aac22d3618223898f"
server_key_version: "encryptionkey@0"
I20260619 13:01:34.245054 10897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 13:01:34.248221 10911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:34.249423 10897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:01:34.249595 10897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "50a46613c7f648159d9cbba0b387c9bf"
format_stamp: "Formatted at 2026-06-19 13:01:34 on dist-test-slave-tpsm"
server_key: "fc71d3cea4727b72a9a350eb43fd6f23"
server_key_iv: "95ed87d252f3105aac22d3618223898f"
server_key_version: "encryptionkey@0"
I20260619 13:01:34.249796 10897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:34.266439 10897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:34.267271 10897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:34.267634 10897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:34.268481 10897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:34.269834 10897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:34.269954 10897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:34.270069 10897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:34.270153 10897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:34.309106 10897 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:34969
I20260619 13:01:34.309157 11023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:34969 every 8 connection(s)
I20260619 13:01:34.310667 10897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:01:34.313670 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 10897
I20260619 13:01:34.313844 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:01:34.314393 1373 external_mini_cluster.cc:1428] Setting key d65bf9e48e58515883897ac169d74509
I20260619 13:01:34.317509 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37409
--builtin_ntp_servers=127.1.87.84:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 13:01:34.324110 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:34.313218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:54787 (local address 127.1.87.126:37409)
0619 13:01:34.313397 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:34.313404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:34.315050 (+ 1646us) server_negotiation.cc:407] Connection header received
0619 13:01:34.317048 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:34.317055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:34.317150 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:34.317263 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:34.318342 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.319546 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.321112 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.321792 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.321935 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:34.323272 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:34.323281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:34.323288 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:34.323357 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:34.323399 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:34.323417 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:34.323503 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:34.323757 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 13:01:34.323867 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:01:34.325552 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:34.313809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37409 (local address 127.1.87.66:54787)
0619 13:01:34.314891 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:34.314938 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:34.316765 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:34.317455 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:34.317464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:34.317546 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:34.318171 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:34.318195 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.319711 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.319715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:34.320959 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:34.320974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.322013 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.322018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:34.322229 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:34.322956 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:01:34.322959 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:34.322998 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:01:34.323082 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:34.323541 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:34.323550 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:34.323557 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:01:34.323718 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 13:01:34.324169 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":315,"threads_started":1}
I20260619 13:01:34.326938 11024 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37409
I20260619 13:01:34.327263 11024 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:34.328056 11024 heartbeater.cc:511] Master 127.1.87.126:37409 requested a full tablet report, sending...
I20260619 13:01:34.329533 10707 ts_manager.cc:194] Registered new tserver with Master: 50a46613c7f648159d9cbba0b387c9bf (127.1.87.66:34969)
I20260619 13:01:34.330415 10707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.66:54787
W20260619 13:01:34.643639 11027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:34.643956 11027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:34.644110 11027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:34.651410 11027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:34.651556 11027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:34.651726 11027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:01:34.661197 11027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:34.662497 11027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:34.664263 11027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:34.677179 11032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:34.677206 11035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:34.677232 11033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:34.678006 11027 server_base.cc:1034] running on GCE node
I20260619 13:01:34.678586 11027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:34.679499 11027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:34.680701 11027 hybrid_clock.cc:648] HybridClock initialized: now 1781874094680653 us; error 41 us; skew 500 ppm
I20260619 13:01:34.681025 11027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:01:34.683287 11027 webserver.cc:460] Webserver started at http://127.1.87.67:38975/ using document root <none> and password file <none>
I20260619 13:01:34.684008 11027 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:34.684119 11027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:34.684441 11027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:34.687227 11027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "48262b59faaf4172917516cb3d383f39"
format_stamp: "Formatted at 2026-06-19 13:01:34 on dist-test-slave-tpsm"
server_key: "6068d9fcc807c1469e0e415ee8d10391"
server_key_iv: "68276e5f009721e4fdab71a714545049"
server_key_version: "encryptionkey@0"
I20260619 13:01:34.688077 11027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "48262b59faaf4172917516cb3d383f39"
format_stamp: "Formatted at 2026-06-19 13:01:34 on dist-test-slave-tpsm"
server_key: "6068d9fcc807c1469e0e415ee8d10391"
server_key_iv: "68276e5f009721e4fdab71a714545049"
server_key_version: "encryptionkey@0"
I20260619 13:01:34.693104 11027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:01:34.696549 11041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:34.697882 11027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:01:34.698050 11027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "48262b59faaf4172917516cb3d383f39"
format_stamp: "Formatted at 2026-06-19 13:01:34 on dist-test-slave-tpsm"
server_key: "6068d9fcc807c1469e0e415ee8d10391"
server_key_iv: "68276e5f009721e4fdab71a714545049"
server_key_version: "encryptionkey@0"
I20260619 13:01:34.698271 11027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:34.727476 11027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:34.728341 11027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:34.728606 11027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:34.729584 11027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:34.731103 11027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:34.731207 11027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:34.731350 11027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:34.731427 11027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 13:01:34.764376 11027 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:45215
I20260619 13:01:34.764472 11153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:45215 every 8 connection(s)
I20260619 13:01:34.765666 11027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:01:34.771956 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 11027
I20260619 13:01:34.772115 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:01:34.772598 1373 external_mini_cluster.cc:1428] Setting key 4a42f3d6e22deb6cb4246b74c2fb29bb
I20260619 13:01:34.778080 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:34.768180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:42583 (local address 127.1.87.126:37409)
0619 13:01:34.768348 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:34.768352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:34.769641 (+ 1289us) server_negotiation.cc:407] Connection header received
0619 13:01:34.771278 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:34.771282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:34.771402 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:34.771490 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:34.772587 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.773597 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.775721 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.776116 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.776227 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:34.777463 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:34.777467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:34.777470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:34.777487 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:34.777515 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:34.777527 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:34.777596 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:34.777784 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 13:01:34.777853 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:01:34.779050 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:34.768676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37409 (local address 127.1.87.67:42583)
0619 13:01:34.769500 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:34.769538 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:34.771046 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:34.771677 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:34.771694 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:34.771773 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:34.772400 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:34.772417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.773755 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.773759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:34.775576 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:34.775587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.776292 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.776296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:34.776478 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:34.777181 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:01:34.777183 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:34.777223 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:01:34.777297 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:34.777606 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:34.777613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:34.777618 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:01:34.777760 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 13:01:34.778029 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":254,"threads_started":1}
I20260619 13:01:34.780393 11154 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37409
I20260619 13:01:34.780639 11154 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:34.781342 11154 heartbeater.cc:511] Master 127.1.87.126:37409 requested a full tablet report, sending...
I20260619 13:01:34.782678 10707 ts_manager.cc:194] Registered new tserver with Master: 48262b59faaf4172917516cb3d383f39 (127.1.87.67:45215)
I20260619 13:01:34.783507 10707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.67:42583
I20260619 13:01:34.787598 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 13:01:34.793424 10895 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:34.791898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.1.87.126:37409)
0619 13:01:34.792080 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:34.792085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:34.792539 (+ 454us) server_negotiation.cc:407] Connection header received
0619 13:01:34.792677 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:34.792682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:34.792752 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:34.792838 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:34.793146 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49334: BlockingRecv error: recv got EOF from 127.0.0.1:49334 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 13:01:34.795118 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10766
I20260619 13:01:34.803274 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:34.796659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.1.87.65:43327)
0619 13:01:34.797088 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:34.797095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:34.797123 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:01:34.797178 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:34.797183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:34.797400 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:34.797523 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:34.797867 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.798804 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.800077 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.800630 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.800732 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:34.801771 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:34.801778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:34.801791 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:34.802413 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:34.802459 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:34.802500 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:34.802603 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:34.802725 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 13:01:34.802908 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":205,"threads_started":1}
I20260619 13:01:34.804702 10888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:34.884227 10893 heartbeater.cc:503] Master 127.1.87.126:37409 was elected leader, sending a full tablet report...
I20260619 13:01:34.885658 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 10766
W20260619 13:01:34.901741 10693 connection.cc:535] server connection from 127.1.87.65:58797 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 13:01:34.902076 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10897
I20260619 13:01:34.910156 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:34.902927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.1.87.66:34969)
0619 13:01:34.903492 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:34.903501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:34.903527 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:01:34.903585 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:34.903591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:34.903804 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:34.903932 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:34.904300 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.905738 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.906917 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:34.907600 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:34.907699 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:34.908739 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:34.908747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:34.908759 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:34.909379 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:34.909423 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:34.909442 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:34.909528 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:34.909662 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 13:01:34.909811 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":268,"threads_started":1}
I20260619 13:01:34.911404 11019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:34.991740 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 10897
I20260619 13:01:35.007719 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 11027
I20260619 13:01:35.015904 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:35.008682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.1.87.67:45215)
0619 13:01:35.009295 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:35.009302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:35.009334 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:01:35.009393 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:35.009399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:35.009610 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:35.009748 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:35.010123 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:35.011109 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:35.012489 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:35.013034 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:35.013138 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:35.014076 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:35.014090 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:35.014110 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:35.014952 (+ 842us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:35.015035 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:35.015063 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:35.015194 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:35.015378 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 13:01:35.015538 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":304,"threads_started":1}
I20260619 13:01:35.017903 11149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:35.100447 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 11027
I20260619 13:01:35.115480 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 10676
I20260619 13:01:35.116530 10737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:35.215147 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 10676
2026-06-19T13:01:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2410 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T13:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:01:35Z Disabled control of system clock
I20260619 13:01:35.282493 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44501
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:44501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 13:01:35.597532 11173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:35.597815 11173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:35.597899 11173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:35.604936 11173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:01:35.605032 11173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:35.605083 11173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:35.605139 11173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:01:35.605181 11173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:01:35.614017 11173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:44501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:35.615214 11173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:35.616874 11173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:35.627861 11178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:35.627934 11179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:35.627899 11181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:35.628978 11173 server_base.cc:1034] running on GCE node
I20260619 13:01:35.629642 11173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:35.630873 11173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:35.632055 11173 hybrid_clock.cc:648] HybridClock initialized: now 1781874095632000 us; error 52 us; skew 500 ppm
I20260619 13:01:35.632339 11173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:01:35.634357 11173 webserver.cc:460] Webserver started at http://127.1.87.126:38231/ using document root <none> and password file <none>
I20260619 13:01:35.634965 11173 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:35.635044 11173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:35.635298 11173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:35.637984 11173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "66c47aeebc3744cdae96482fc551b6e1"
format_stamp: "Formatted at 2026-06-19 13:01:35 on dist-test-slave-tpsm"
server_key: "d6da0bedb467eac16886507c43c114db"
server_key_iv: "04375d71b15c32fcf1e5a3030bedc4aa"
server_key_version: "encryptionkey@0"
I20260619 13:01:35.638681 11173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "66c47aeebc3744cdae96482fc551b6e1"
format_stamp: "Formatted at 2026-06-19 13:01:35 on dist-test-slave-tpsm"
server_key: "d6da0bedb467eac16886507c43c114db"
server_key_iv: "04375d71b15c32fcf1e5a3030bedc4aa"
server_key_version: "encryptionkey@0"
I20260619 13:01:35.643234 11173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 13:01:35.646296 11187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:35.647542 11173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:01:35.647671 11173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "66c47aeebc3744cdae96482fc551b6e1"
format_stamp: "Formatted at 2026-06-19 13:01:35 on dist-test-slave-tpsm"
server_key: "d6da0bedb467eac16886507c43c114db"
server_key_iv: "04375d71b15c32fcf1e5a3030bedc4aa"
server_key_version: "encryptionkey@0"
I20260619 13:01:35.647833 11173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:35.653056 11173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:35.653715 11173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:35.653954 11173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:35.671330 11173 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:44501
I20260619 13:01:35.671406 11238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:44501 every 8 connection(s)
I20260619 13:01:35.672714 11173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:01:35.676157 11239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:35.680814 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 11173
I20260619 13:01:35.681022 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:01:35.681648 1373 external_mini_cluster.cc:1428] Setting key fcf021c79e4dc0eb42ac7a5669eb3ef1
I20260619 13:01:35.687227 11239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1: Bootstrap starting.
I20260619 13:01:35.690580 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:35.683564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.1.87.126:44501)
0619 13:01:35.684643 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:35.684656 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:35.684706 (+ 50us) server_negotiation.cc:407] Connection header received
0619 13:01:35.686916 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:35.686943 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:35.687430 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:35.687830 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:35.688762 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:35.688770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:35.688791 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:35.688810 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:35.688897 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:35.688923 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:35.689139 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:35.689279 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 13:01:35.689525 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":347,"threads_started":1}
I20260619 13:01:35.690934 11239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:35.691947 11239 log.cc:824] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:35.694501 11239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1: No bootstrap required, opened a new log
I20260619 13:01:35.701185 11239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c47aeebc3744cdae96482fc551b6e1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44501 } }
I20260619 13:01:35.701534 11239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:35.701642 11239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c47aeebc3744cdae96482fc551b6e1, State: Initialized, Role: FOLLOWER
I20260619 13:01:35.702175 11239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c47aeebc3744cdae96482fc551b6e1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44501 } }
I20260619 13:01:35.702399 11239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:01:35.702536 11239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:01:35.702692 11239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:35.704842 11239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c47aeebc3744cdae96482fc551b6e1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44501 } }
I20260619 13:01:35.705278 11239 leader_election.cc:302] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66c47aeebc3744cdae96482fc551b6e1; no voters:
I20260619 13:01:35.705792 11239 leader_election.cc:288] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:01:35.705925 11244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:35.706655 11244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [term 1 LEADER]: Becoming Leader. State: Replica: 66c47aeebc3744cdae96482fc551b6e1, State: Running, Role: LEADER
I20260619 13:01:35.707193 11244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c47aeebc3744cdae96482fc551b6e1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44501 } }
I20260619 13:01:35.707722 11239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:01:35.709836 11246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c47aeebc3744cdae96482fc551b6e1. Latest consensus state: current_term: 1 leader_uuid: "66c47aeebc3744cdae96482fc551b6e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c47aeebc3744cdae96482fc551b6e1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44501 } } }
I20260619 13:01:35.710155 11246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:35.710968 11245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66c47aeebc3744cdae96482fc551b6e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c47aeebc3744cdae96482fc551b6e1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44501 } } }
I20260619 13:01:35.711362 11245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:35.717499 11250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:01:35.723511 11250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:01:35.731079 11250 catalog_manager.cc:1347] Generated new cluster ID: ada43e481c7248dd80e2f6ffa9fe77ba
I20260619 13:01:35.731178 11250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:01:35.755815 11250 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:01:35.756253 11250 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:01:35.764071 11250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66c47aeebc3744cdae96482fc551b6e1: Generated new TSK 0
I20260619 13:01:35.764991 11250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:01:35.783749 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44501
--builtin_ntp_servers=127.1.87.84:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 13:01:36.100953 11263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:36.101210 11263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:36.101353 11263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:36.108821 11263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:36.108948 11263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:36.109135 11263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:01:36.118220 11263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:36.119421 11263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:36.121294 11263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:36.133944 11271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:36.134012 11268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:36.133984 11269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:36.134698 11263 server_base.cc:1034] running on GCE node
I20260619 13:01:36.135286 11263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:36.136247 11263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:36.137461 11263 hybrid_clock.cc:648] HybridClock initialized: now 1781874096137410 us; error 41 us; skew 500 ppm
I20260619 13:01:36.137795 11263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:01:36.139998 11263 webserver.cc:460] Webserver started at http://127.1.87.65:34297/ using document root <none> and password file <none>
I20260619 13:01:36.140631 11263 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:36.140760 11263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:36.141070 11263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:36.143772 11263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "62d317cbc024470d99ff2183687b263a"
format_stamp: "Formatted at 2026-06-19 13:01:36 on dist-test-slave-tpsm"
server_key: "43a50c4a1f11bee577eb36a780022789"
server_key_iv: "44858907270905539e5be317eb782e17"
server_key_version: "encryptionkey@0"
I20260619 13:01:36.144510 11263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "62d317cbc024470d99ff2183687b263a"
format_stamp: "Formatted at 2026-06-19 13:01:36 on dist-test-slave-tpsm"
server_key: "43a50c4a1f11bee577eb36a780022789"
server_key_iv: "44858907270905539e5be317eb782e17"
server_key_version: "encryptionkey@0"
I20260619 13:01:36.149397 11263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:01:36.152791 11277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:36.154058 11263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:01:36.154208 11263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "62d317cbc024470d99ff2183687b263a"
format_stamp: "Formatted at 2026-06-19 13:01:36 on dist-test-slave-tpsm"
server_key: "43a50c4a1f11bee577eb36a780022789"
server_key_iv: "44858907270905539e5be317eb782e17"
server_key_version: "encryptionkey@0"
I20260619 13:01:36.154413 11263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:36.159844 11263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:36.160576 11263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:36.160843 11263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:36.161684 11263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:36.162966 11263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:36.163076 11263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:36.163200 11263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:36.163288 11263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:36.194440 11263 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:37551
I20260619 13:01:36.194507 11389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:37551 every 8 connection(s)
I20260619 13:01:36.195778 11263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:01:36.203542 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:36.198159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:43641 (local address 127.1.87.126:44501)
0619 13:01:36.198534 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:36.198538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:36.199744 (+ 1206us) server_negotiation.cc:407] Connection header received
0619 13:01:36.201363 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:36.201369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:36.201443 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:36.201554 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:36.202792 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:36.202797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:36.202802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:36.202820 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:36.202867 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:36.202879 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:36.203001 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:36.203174 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 13:01:36.203239 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":186,"threads_started":1}
I20260619 13:01:36.204205 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 11263
I20260619 13:01:36.204412 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:01:36.204721 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:36.198732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44501 (local address 127.1.87.65:43641)
0619 13:01:36.199599 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:36.199638 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:36.201102 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:36.201767 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:36.201775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:36.201858 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:36.202516 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:01:36.202519 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:36.202554 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:01:36.202630 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:36.202968 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:36.202979 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:36.202988 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 13:01:36.203266 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 13:01:36.203522 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":252,"threads_started":1}
I20260619 13:01:36.205197 1373 external_mini_cluster.cc:1428] Setting key 698f2660353b94cf5dc11c8daa280da3
I20260619 13:01:36.206184 11390 heartbeater.cc:347] Connected to a master server at 127.1.87.126:44501
I20260619 13:01:36.206487 11390 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:36.207132 11390 heartbeater.cc:511] Master 127.1.87.126:44501 requested a full tablet report, sending...
I20260619 13:01:36.207897 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44501
--builtin_ntp_servers=127.1.87.84:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 13:01:36.209158 11204 ts_manager.cc:194] Registered new tserver with Master: 62d317cbc024470d99ff2183687b263a (127.1.87.65:37551)
W20260619 13:01:36.531390 11394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:36.531735 11394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:36.531862 11394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:36.539166 11394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:36.539270 11394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:36.539451 11394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:01:36.548764 11394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:36.549939 11394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:36.551659 11394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:36.565732 11399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:36.565594 11402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:36.565551 11400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:36.566704 11394 server_base.cc:1034] running on GCE node
I20260619 13:01:36.567394 11394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:36.568390 11394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:36.569608 11394 hybrid_clock.cc:648] HybridClock initialized: now 1781874096569576 us; error 57 us; skew 500 ppm
I20260619 13:01:36.569924 11394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:01:36.572136 11394 webserver.cc:460] Webserver started at http://127.1.87.66:32799/ using document root <none> and password file <none>
I20260619 13:01:36.572824 11394 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:36.572947 11394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:36.573288 11394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:36.576205 11394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "9fafab1231574d8d8b9f7ad6f69b40a2"
format_stamp: "Formatted at 2026-06-19 13:01:36 on dist-test-slave-tpsm"
server_key: "78b1f000e7d75bdef61eaeccfdd8226d"
server_key_iv: "cdd2f8206a926e2c44f15d39da3b0560"
server_key_version: "encryptionkey@0"
I20260619 13:01:36.576961 11394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "9fafab1231574d8d8b9f7ad6f69b40a2"
format_stamp: "Formatted at 2026-06-19 13:01:36 on dist-test-slave-tpsm"
server_key: "78b1f000e7d75bdef61eaeccfdd8226d"
server_key_iv: "cdd2f8206a926e2c44f15d39da3b0560"
server_key_version: "encryptionkey@0"
I20260619 13:01:36.581909 11394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:01:36.585069 11408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:36.586269 11394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:01:36.586428 11394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "9fafab1231574d8d8b9f7ad6f69b40a2"
format_stamp: "Formatted at 2026-06-19 13:01:36 on dist-test-slave-tpsm"
server_key: "78b1f000e7d75bdef61eaeccfdd8226d"
server_key_iv: "cdd2f8206a926e2c44f15d39da3b0560"
server_key_version: "encryptionkey@0"
I20260619 13:01:36.586650 11394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:36.592324 11394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:36.593067 11394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:36.593323 11394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:36.594175 11394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:36.595759 11394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:36.595880 11394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:36.596002 11394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:36.596087 11394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:36.628312 11394 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:40361
I20260619 13:01:36.628399 11520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:40361 every 8 connection(s)
I20260619 13:01:36.629632 11394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:01:36.637115 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:36.631999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:50237 (local address 127.1.87.126:44501)
0619 13:01:36.632213 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:36.632219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:36.633552 (+ 1333us) server_negotiation.cc:407] Connection header received
0619 13:01:36.635117 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:36.635121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:36.635185 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:36.635274 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:36.636556 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:36.636560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:36.636563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:36.636577 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:36.636607 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:36.636617 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:36.636718 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:36.636864 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 13:01:36.636911 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 13:01:36.637964 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:36.632525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44501 (local address 127.1.87.66:50237)
0619 13:01:36.633388 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:36.633433 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:36.634882 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:36.635519 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:36.635527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:36.635608 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:36.636284 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:01:36.636286 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:36.636321 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:01:36.636400 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:36.636702 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:36.636708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:36.636713 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:01:36.636911 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 13:01:36.637082 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":272,"threads_started":1}
I20260619 13:01:36.639252 11521 heartbeater.cc:347] Connected to a master server at 127.1.87.126:44501
I20260619 13:01:36.639273 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 11394
I20260619 13:01:36.639477 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:01:36.639576 11521 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:36.640007 1373 external_mini_cluster.cc:1428] Setting key 529bda2acdfd71f4dc3484e6d7f20847
I20260619 13:01:36.640254 11521 heartbeater.cc:511] Master 127.1.87.126:44501 requested a full tablet report, sending...
I20260619 13:01:36.641680 11203 ts_manager.cc:194] Registered new tserver with Master: 9fafab1231574d8d8b9f7ad6f69b40a2 (127.1.87.66:40361)
I20260619 13:01:36.642903 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44501
--builtin_ntp_servers=127.1.87.84:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 13:01:36.969938 11524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:36.970204 11524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:36.970371 11524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:36.977550 11524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:36.977649 11524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:36.977815 11524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:01:36.986792 11524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:36.988125 11524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:36.989966 11524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:37.002861 11530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:37.002947 11532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:37.002967 11529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:37.003549 11524 server_base.cc:1034] running on GCE node
I20260619 13:01:37.004081 11524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:37.004971 11524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:37.006225 11524 hybrid_clock.cc:648] HybridClock initialized: now 1781874097006161 us; error 57 us; skew 500 ppm
I20260619 13:01:37.006498 11524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:01:37.008636 11524 webserver.cc:460] Webserver started at http://127.1.87.67:36993/ using document root <none> and password file <none>
I20260619 13:01:37.009301 11524 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:37.009388 11524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:37.009652 11524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:37.012398 11524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "fda1c6914fed4daa84075fc127af3f46"
format_stamp: "Formatted at 2026-06-19 13:01:37 on dist-test-slave-tpsm"
server_key: "c75e701a26d825e1344858908733c8f5"
server_key_iv: "7ca12ee9cf0fc812b8b44bc16cc98e21"
server_key_version: "encryptionkey@0"
I20260619 13:01:37.013123 11524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "fda1c6914fed4daa84075fc127af3f46"
format_stamp: "Formatted at 2026-06-19 13:01:37 on dist-test-slave-tpsm"
server_key: "c75e701a26d825e1344858908733c8f5"
server_key_iv: "7ca12ee9cf0fc812b8b44bc16cc98e21"
server_key_version: "encryptionkey@0"
I20260619 13:01:37.018006 11524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:01:37.021345 11538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:37.022766 11524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:01:37.022902 11524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "fda1c6914fed4daa84075fc127af3f46"
format_stamp: "Formatted at 2026-06-19 13:01:37 on dist-test-slave-tpsm"
server_key: "c75e701a26d825e1344858908733c8f5"
server_key_iv: "7ca12ee9cf0fc812b8b44bc16cc98e21"
server_key_version: "encryptionkey@0"
I20260619 13:01:37.023072 11524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:37.028487 11524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:37.029163 11524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:37.029402 11524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:37.030228 11524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:37.031560 11524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:37.031646 11524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:37.031724 11524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:37.031780 11524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:37.065085 11524 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:35515
I20260619 13:01:37.065141 11650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:35515 every 8 connection(s)
I20260619 13:01:37.066378 11524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:01:37.074282 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 11524
I20260619 13:01:37.074333 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:37.068953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:33925 (local address 127.1.87.126:44501)
0619 13:01:37.069128 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:37.069133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:37.070472 (+ 1339us) server_negotiation.cc:407] Connection header received
0619 13:01:37.072149 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:37.072155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:37.072231 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:37.072341 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:37.073594 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:37.073600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:37.073605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:37.073622 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:37.073661 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:37.073672 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:37.073799 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:37.074000 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 13:01:37.074066 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:01:37.074492 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:01:37.074934 1373 external_mini_cluster.cc:1428] Setting key ed745a300cf20fcb1e6272baad19e2df
I20260619 13:01:37.075265 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:37.069448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44501 (local address 127.1.87.67:33925)
0619 13:01:37.070299 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:37.070357 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:37.071898 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:37.072549 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:37.072557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:37.072639 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:37.073320 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:01:37.073322 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:37.073357 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:01:37.073432 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:37.073771 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:37.073781 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:37.073787 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:01:37.074082 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 13:01:37.074322 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":265,"threads_started":1}
I20260619 13:01:37.076825 11651 heartbeater.cc:347] Connected to a master server at 127.1.87.126:44501
I20260619 13:01:37.077106 11651 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:37.077641 11651 heartbeater.cc:511] Master 127.1.87.126:44501 requested a full tablet report, sending...
I20260619 13:01:37.078879 11203 ts_manager.cc:194] Registered new tserver with Master: fda1c6914fed4daa84075fc127af3f46 (127.1.87.67:35515)
I20260619 13:01:37.089948 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 13:01:37.095935 11392 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:01:37.094500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.1.87.126:44501)
0619 13:01:37.094667 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:37.094672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:37.095161 (+ 489us) server_negotiation.cc:407] Connection header received
0619 13:01:37.095299 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:37.095302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:37.095396 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:37.095477 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:37.095752 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39252: BlockingRecv error: recv got EOF from 127.0.0.1:39252 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:01:37.096720 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 11263
I20260619 13:01:37.100135 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:37.097684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.1.87.65:37551)
0619 13:01:37.098209 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:37.098216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:37.098242 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:01:37.098294 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:37.098300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:37.098499 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:37.098618 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:37.099465 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:37.099471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:37.099482 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:37.099499 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:37.099537 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:37.099550 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:37.099651 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:37.099765 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 13:01:37.099874 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":245,"threads_started":1}
I20260619 13:01:37.101358 11385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:37.181232 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 11263
I20260619 13:01:37.197749 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 11394
I20260619 13:01:37.201200 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:37.198624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.1.87.66:40361)
0619 13:01:37.199079 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:37.199085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:37.199110 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:01:37.199177 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:37.199183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:37.199441 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:37.199603 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:37.200538 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:37.200544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:37.200554 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:37.200571 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:37.200606 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:37.200619 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:37.200765 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:37.200844 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 13:01:37.200936 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":209,"threads_started":1}
I20260619 13:01:37.202346 11516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:37.283020 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 11394
I20260619 13:01:37.298444 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 11524
I20260619 13:01:37.302366 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:37.299456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.1.87.67:35515)
0619 13:01:37.300059 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:37.300066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:37.300094 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:01:37.300152 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:37.300158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:37.300374 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:37.300538 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:37.301657 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:37.301663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:37.301673 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:01:37.301684 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:37.301720 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:01:37.301733 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:37.301913 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:37.301986 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 13:01:37.302078 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":326,"threads_started":1}
I20260619 13:01:37.303570 11646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:37.384007 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 11524
I20260619 13:01:37.399868 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 11173
I20260619 13:01:37.400959 11234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:37.499886 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 11173
2026-06-19T13:01:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2285 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:01:37 dist-test-slave-tpsm krb5kdc[11669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:01:37 dist-test-slave-tpsm krb5kdc[11669](info): set up 2 sockets
Jun 19 13:01:37 dist-test-slave-tpsm krb5kdc[11669](info): commencing operation
krb5kdc: starting...
W20260619 13:01:38.600821 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:01:38 dist-test-slave-tpsm krb5kdc[11669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:01:38.826661 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:43621
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:39.148202 11682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:39.148479 11682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:39.148571 11682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:39.155701 11682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:01:39.155800 11682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:39.155845 11682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:39.155901 11682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:01:39.155939 11682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:01:39.165730 11682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:43621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:39.167035 11682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:39.168278 11682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:39.179705 11687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:39.179891 11690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:39.179901 11688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:39.180984 11682 server_base.cc:1034] running on GCE node
I20260619 13:01:39.181651 11682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:39.182955 11682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:39.184147 11682 hybrid_clock.cc:648] HybridClock initialized: now 1781874099184104 us; error 44 us; skew 500 ppm
I20260619 13:01:39.184437 11682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:39 dist-test-slave-tpsm krb5kdc[11669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874099, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:39.193742 11682 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:01:39.195458 11682 webserver.cc:460] Webserver started at http://127.1.87.126:42995/ using document root <none> and password file <none>
I20260619 13:01:39.196173 11682 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:39.196328 11682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:39.196650 11682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:39.199726 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "a8138562ba164c6d9164ad35d5f5b8dc"
format_stamp: "Formatted at 2026-06-19 13:01:39 on dist-test-slave-tpsm"
server_key: "206478eaf08d07bbb3f509114a8ef26f"
server_key_iv: "7c79609fa0d09081ca7db6dfbcf84983"
server_key_version: "encryptionkey@0"
I20260619 13:01:39.200531 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "a8138562ba164c6d9164ad35d5f5b8dc"
format_stamp: "Formatted at 2026-06-19 13:01:39 on dist-test-slave-tpsm"
server_key: "206478eaf08d07bbb3f509114a8ef26f"
server_key_iv: "7c79609fa0d09081ca7db6dfbcf84983"
server_key_version: "encryptionkey@0"
I20260619 13:01:39.205857 11682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 13:01:39.209484 11697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:39.210920 11682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:01:39.211073 11682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "a8138562ba164c6d9164ad35d5f5b8dc"
format_stamp: "Formatted at 2026-06-19 13:01:39 on dist-test-slave-tpsm"
server_key: "206478eaf08d07bbb3f509114a8ef26f"
server_key_iv: "7c79609fa0d09081ca7db6dfbcf84983"
server_key_version: "encryptionkey@0"
I20260619 13:01:39.211241 11682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:39.224437 11682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:39.228480 11682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:39.228747 11682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:39.247817 11682 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:43621
I20260619 13:01:39.247934 11748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:43621 every 8 connection(s)
I20260619 13:01:39.249279 11682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:01:39.252761 11749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:39.258293 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 11682
I20260619 13:01:39.258538 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:01:39.259135 1373 external_mini_cluster.cc:1428] Setting key 0a4e52c0daa72d9199df233b60a4d845
I20260619 13:01:39.263266 11749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc: Bootstrap starting.
I20260619 13:01:39.266892 11749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:39.268025 11749 log.cc:824] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:39.270949 11749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc: No bootstrap required, opened a new log
Jun 19 13:01:39 dist-test-slave-tpsm krb5kdc[11669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:01:39.279003 11749 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8138562ba164c6d9164ad35d5f5b8dc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43621 } }
I20260619 13:01:39.279438 11749 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:39.279567 11749 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8138562ba164c6d9164ad35d5f5b8dc, State: Initialized, Role: FOLLOWER
I20260619 13:01:39.280241 11749 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8138562ba164c6d9164ad35d5f5b8dc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43621 } }
I20260619 13:01:39.280462 11749 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:01:39.280649 11749 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:01:39.280848 11749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:39.282711 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:39.261129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.1.87.126:43621)
0619 13:01:39.261908 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:39.261919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:39.261955 (+ 36us) server_negotiation.cc:407] Connection header received
0619 13:01:39.264144 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:39.264165 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:39.264508 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:39.264829 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:39.266601 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:39.267993 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:39.269589 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:39.270247 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:39.270374 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:39.274817 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:39.274825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:39.274838 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:39.274877 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:39.278005 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:39.278701 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:39.278716 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:39.278724 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:39.278881 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:39.279263 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:39.279270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:39.279278 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:39.279980 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:39.280296 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:39.280869 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 13:01:39.281310 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":352,"threads_started":1}
I20260619 13:01:39.283840 11749 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8138562ba164c6d9164ad35d5f5b8dc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43621 } }
I20260619 13:01:39.284336 11749 leader_election.cc:302] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8138562ba164c6d9164ad35d5f5b8dc; no voters:
I20260619 13:01:39.284997 11749 leader_election.cc:288] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:01:39.285109 11754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:39.285943 11754 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [term 1 LEADER]: Becoming Leader. State: Replica: a8138562ba164c6d9164ad35d5f5b8dc, State: Running, Role: LEADER
I20260619 13:01:39.286540 11754 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8138562ba164c6d9164ad35d5f5b8dc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43621 } }
I20260619 13:01:39.287520 11749 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:01:39.289278 11755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8138562ba164c6d9164ad35d5f5b8dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8138562ba164c6d9164ad35d5f5b8dc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43621 } } }
I20260619 13:01:39.289623 11755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:39.291558 11756 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8138562ba164c6d9164ad35d5f5b8dc. Latest consensus state: current_term: 1 leader_uuid: "a8138562ba164c6d9164ad35d5f5b8dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8138562ba164c6d9164ad35d5f5b8dc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 43621 } } }
I20260619 13:01:39.291853 11756 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc [sys.catalog]: This master's current role is: LEADER
I20260619 13:01:39.296538 11762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:01:39.301785 11762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:01:39.310554 11762 catalog_manager.cc:1347] Generated new cluster ID: 3086580fa0fb40ceb30580fa42f96b20
I20260619 13:01:39.310693 11762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:01:39.331857 11762 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:01:39.333690 11762 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:01:39.354771 11762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc: Generated new TSK 0
I20260619 13:01:39.355782 11762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:01:39.420351 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43621
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:39.748953 11775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:39.749218 11775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:39.749397 11775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:39.756407 11775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:39.756512 11775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:39.756734 11775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:01:39.766217 11775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:39.767455 11775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:39.768695 11775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:39.781183 11781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:39.781361 11783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:39.781250 11780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:39.782303 11775 server_base.cc:1034] running on GCE node
I20260619 13:01:39.782925 11775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:39.784086 11775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:39.785332 11775 hybrid_clock.cc:648] HybridClock initialized: now 1781874099785281 us; error 38 us; skew 500 ppm
I20260619 13:01:39.785597 11775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:39 dist-test-slave-tpsm krb5kdc[11669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874099, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:39.788785 11775 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:01:39.790135 11775 webserver.cc:460] Webserver started at http://127.1.87.65:45699/ using document root <none> and password file <none>
I20260619 13:01:39.790791 11775 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:39.790922 11775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:39.791296 11775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:39.794493 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "6faa4ca72ba64e91a82c098000f103fc"
format_stamp: "Formatted at 2026-06-19 13:01:39 on dist-test-slave-tpsm"
server_key: "3b708566f585443fc27883620a3fe022"
server_key_iv: "2ee17207dad7b84dc4b0176739b60fd7"
server_key_version: "encryptionkey@0"
I20260619 13:01:39.795260 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "6faa4ca72ba64e91a82c098000f103fc"
format_stamp: "Formatted at 2026-06-19 13:01:39 on dist-test-slave-tpsm"
server_key: "3b708566f585443fc27883620a3fe022"
server_key_iv: "2ee17207dad7b84dc4b0176739b60fd7"
server_key_version: "encryptionkey@0"
I20260619 13:01:39.800357 11775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:01:39.803802 11790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:39.805169 11775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:01:39.805368 11775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "6faa4ca72ba64e91a82c098000f103fc"
format_stamp: "Formatted at 2026-06-19 13:01:39 on dist-test-slave-tpsm"
server_key: "3b708566f585443fc27883620a3fe022"
server_key_iv: "2ee17207dad7b84dc4b0176739b60fd7"
server_key_version: "encryptionkey@0"
I20260619 13:01:39.805542 11775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:39.824343 11775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:39.828964 11775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:39.829377 11775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:39.830396 11775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:39.831897 11775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:39.831971 11775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:39.832137 11775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:39.832199 11775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:39.866783 11775 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:41255
I20260619 13:01:39.866827 11902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:41255 every 8 connection(s)
I20260619 13:01:39.868101 11775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:01:39.875746 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 11775
I20260619 13:01:39.875967 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:01:39.876578 1373 external_mini_cluster.cc:1428] Setting key 115aaf4cdfaf6e15e852a9482015ca08
Jun 19 13:01:39 dist-test-slave-tpsm krb5kdc[11669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874099, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:01:39.887485 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:39.870652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:34199 (local address 127.1.87.126:43621)
0619 13:01:39.871052 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:39.871057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:39.872217 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 13:01:39.873965 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:39.873971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:39.874061 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:39.874201 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:39.875691 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:39.876542 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:39.878214 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:39.878594 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:39.878731 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:39.882029 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:39.882035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:39.882040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:39.882080 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:39.884512 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:39.885233 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:39.885239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:39.885242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:39.885360 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:39.885786 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:39.885790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:39.885794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:39.886043 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:39.886203 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:39.886866 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 13:01:39.887084 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":202,"threads_started":1}
I20260619 13:01:39.888387 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:39.871117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:43621 (local address 127.1.87.65:34199)
0619 13:01:39.872046 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:39.872082 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:39.873705 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:39.874422 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:39.874429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:39.874919 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:39.875527 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:39.875545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:39.876701 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:39.876713 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:39.878049 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:39.878060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:39.878828 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:39.878835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:39.879055 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:39.879755 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:39.879780 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:39.881827 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:39.884656 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:39.884664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:39.884681 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:39.885099 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:39.885469 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:39.885474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:39.885478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:39.885660 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:39.886168 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:39.886176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:39.886498 (+ 322us) client_negotiation.cc:769] Sending connection context
0619 13:01:39.886763 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 13:01:39.887283 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":392,"threads_started":1}
I20260619 13:01:39.889951 11903 heartbeater.cc:347] Connected to a master server at 127.1.87.126:43621
I20260619 13:01:39.890249 11903 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:39.890980 11903 heartbeater.cc:511] Master 127.1.87.126:43621 requested a full tablet report, sending...
I20260619 13:01:39.893070 11714 ts_manager.cc:194] Registered new tserver with Master: 6faa4ca72ba64e91a82c098000f103fc (127.1.87.65:41255)
I20260619 13:01:39.894500 11714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:34199
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:01:39.924782 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43621
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:40.258088 11909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:40.258427 11909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:40.258564 11909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:40.270141 11909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:40.270272 11909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:40.270494 11909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:01:40.280215 11909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:40.281400 11909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:40.282625 11909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:40.296000 11914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:40.296556 11915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:40.296916 11917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:40.298369 11909 server_base.cc:1034] running on GCE node
I20260619 13:01:40.298942 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:40.299921 11909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:40.301149 11909 hybrid_clock.cc:648] HybridClock initialized: now 1781874100301091 us; error 54 us; skew 500 ppm
I20260619 13:01:40.301445 11909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:40 dist-test-slave-tpsm krb5kdc[11669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874100, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:40.304638 11909 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:01:40.306066 11909 webserver.cc:460] Webserver started at http://127.1.87.66:38137/ using document root <none> and password file <none>
I20260619 13:01:40.306739 11909 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:40.306860 11909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:40.307153 11909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:40.310067 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "edc300b137af4ae0a7a7a1fa9c067850"
format_stamp: "Formatted at 2026-06-19 13:01:40 on dist-test-slave-tpsm"
server_key: "55fb74b0b1d80904aff51ba4924f7435"
server_key_iv: "5a0dc35f9dc0cd1311009585397bd3ec"
server_key_version: "encryptionkey@0"
I20260619 13:01:40.310863 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "edc300b137af4ae0a7a7a1fa9c067850"
format_stamp: "Formatted at 2026-06-19 13:01:40 on dist-test-slave-tpsm"
server_key: "55fb74b0b1d80904aff51ba4924f7435"
server_key_iv: "5a0dc35f9dc0cd1311009585397bd3ec"
server_key_version: "encryptionkey@0"
I20260619 13:01:40.316215 11909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:01:40.319675 11924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:40.321069 11909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:01:40.321233 11909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "edc300b137af4ae0a7a7a1fa9c067850"
format_stamp: "Formatted at 2026-06-19 13:01:40 on dist-test-slave-tpsm"
server_key: "55fb74b0b1d80904aff51ba4924f7435"
server_key_iv: "5a0dc35f9dc0cd1311009585397bd3ec"
server_key_version: "encryptionkey@0"
I20260619 13:01:40.321416 11909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:40.334267 11909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:40.338539 11909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:40.338829 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:40.339907 11909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:40.341308 11909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:40.341382 11909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:40.341533 11909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:40.341586 11909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:40.376206 11909 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:41865
I20260619 13:01:40.376298 12036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:41865 every 8 connection(s)
I20260619 13:01:40.377462 11909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:01:40.378748 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 11909
I20260619 13:01:40.378937 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:01:40.379477 1373 external_mini_cluster.cc:1428] Setting key 7fd15e9a9bf2232e85df318eb8655e1f
Jun 19 13:01:40 dist-test-slave-tpsm krb5kdc[11669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874100, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:01:40.400760 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.380377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:55107 (local address 127.1.87.126:43621)
0619 13:01:40.380575 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:40.380587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:40.381884 (+ 1297us) server_negotiation.cc:407] Connection header received
0619 13:01:40.383717 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:40.383723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:40.383811 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:40.383917 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:40.385324 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.386015 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.387919 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.388735 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.388891 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.393191 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:40.393200 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:40.393208 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:40.393253 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:40.397185 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:40.397997 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:40.398006 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:40.398012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:40.398208 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:40.398685 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:40.398693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:40.398699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:40.399105 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:40.399272 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:40.400060 (+ 788us) server_negotiation.cc:299] Negotiation successful
0619 13:01:40.400354 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:01:40.401568 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.380852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:43621 (local address 127.1.87.66:55107)
0619 13:01:40.381740 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:40.381783 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:40.383450 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:40.384141 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:40.384149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:40.384633 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:40.385166 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.385181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.386186 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.386192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.387749 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.387766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.389012 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.389022 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.389411 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.390229 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:40.390263 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:40.392935 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:40.397382 (+ 4447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:40.397390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:40.397406 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:40.397852 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:40.398343 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:40.398348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:40.398351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:40.398554 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:40.399241 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:40.399247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:40.399675 (+ 428us) client_negotiation.cc:769] Sending connection context
0619 13:01:40.399907 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 13:01:40.400427 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":330,"threads_started":1}
I20260619 13:01:40.403157 12037 heartbeater.cc:347] Connected to a master server at 127.1.87.126:43621
I20260619 13:01:40.403505 12037 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:01:40.404276 12037 heartbeater.cc:511] Master 127.1.87.126:43621 requested a full tablet report, sending...
I20260619 13:01:40.405735 11714 ts_manager.cc:194] Registered new tserver with Master: edc300b137af4ae0a7a7a1fa9c067850 (127.1.87.66:41865)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
I20260619 13:01:40.406641 11714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:55107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:01:40.434528 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43621
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:01:40.760203 12042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:01:40.760488 12042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:01:40.760645 12042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:01:40.767997 12042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:01:40.768110 12042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:01:40.768282 12042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:01:40.778378 12042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:43621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:01:40.779719 12042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:01:40.781014 12042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:01:40.793947 12047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:40.794031 12050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:01:40.793991 12048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:01:40.794787 12042 server_base.cc:1034] running on GCE node
I20260619 13:01:40.795460 12042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:01:40.796629 12042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:01:40.797885 12042 hybrid_clock.cc:648] HybridClock initialized: now 1781874100797818 us; error 55 us; skew 500 ppm
I20260619 13:01:40.798214 12042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:01:40 dist-test-slave-tpsm krb5kdc[11669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874100, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:01:40.801226 12042 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:01:40.802835 12042 webserver.cc:460] Webserver started at http://127.1.87.67:40871/ using document root <none> and password file <none>
I20260619 13:01:40.803577 12042 fs_manager.cc:359] Metadata directory not provided
I20260619 13:01:40.803725 12042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:01:40.804020 12042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:01:40.806912 12042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1"
format_stamp: "Formatted at 2026-06-19 13:01:40 on dist-test-slave-tpsm"
server_key: "34a0998d1670eec9d58ec0ee244801a2"
server_key_iv: "550750ef2b58f7931245373c1ef65ce9"
server_key_version: "encryptionkey@0"
I20260619 13:01:40.807729 12042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1"
format_stamp: "Formatted at 2026-06-19 13:01:40 on dist-test-slave-tpsm"
server_key: "34a0998d1670eec9d58ec0ee244801a2"
server_key_iv: "550750ef2b58f7931245373c1ef65ce9"
server_key_version: "encryptionkey@0"
I20260619 13:01:40.812642 12042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 13:01:40.815924 12057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:40.817190 12042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:01:40.817358 12042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1"
format_stamp: "Formatted at 2026-06-19 13:01:40 on dist-test-slave-tpsm"
server_key: "34a0998d1670eec9d58ec0ee244801a2"
server_key_iv: "550750ef2b58f7931245373c1ef65ce9"
server_key_version: "encryptionkey@0"
I20260619 13:01:40.817555 12042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:01:40.852066 12042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:01:40.855892 12042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:01:40.856182 12042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:01:40.857136 12042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:01:40.858433 12042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:01:40.858536 12042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:40.858692 12042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:01:40.858771 12042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:01:40.891912 12042 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:39403
I20260619 13:01:40.891957 12169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:39403 every 8 connection(s)
I20260619 13:01:40.893148 12042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:01:40.897760 11903 heartbeater.cc:503] Master 127.1.87.126:43621 was elected leader, sending a full tablet report...
I20260619 13:01:40.897884 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 12042
I20260619 13:01:40.898070 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:01:40.898604 1373 external_mini_cluster.cc:1428] Setting key 1e8ab3a73c5ac4e3ffa4eac40e622b88
Jun 19 13:01:40 dist-test-slave-tpsm krb5kdc[11669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874100, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:01:40.914290 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.895531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:50303 (local address 127.1.87.126:43621)
0619 13:01:40.895728 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:40.895734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:40.896920 (+ 1186us) server_negotiation.cc:407] Connection header received
0619 13:01:40.899198 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:40.899203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:40.899291 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:40.899428 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:40.901540 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.902451 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.904187 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.904603 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.904710 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.908322 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:40.908327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:40.908330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:40.908363 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:40.911024 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:40.911814 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:40.911826 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:40.911829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:40.911962 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:40.912383 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:40.912388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:40.912391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:40.912703 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:40.912853 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:40.913704 (+ 851us) server_negotiation.cc:299] Negotiation successful
0619 13:01:40.913951 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:01:40.915230 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.896021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:43621 (local address 127.1.87.67:50303)
0619 13:01:40.896751 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:40.896800 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:40.898813 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:40.899795 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:40.899808 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:40.900566 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:01:40.901349 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.901378 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.902612 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.902616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.904016 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.904028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.904813 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.904818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.905004 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.905699 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:01:40.905722 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:01:40.908130 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:01:40.911187 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:40.911197 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:40.911216 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:40.911686 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:40.912087 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:01:40.912092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:01:40.912096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:01:40.912268 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:01:40.912853 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:01:40.912863 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:01:40.913225 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 13:01:40.913536 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 13:01:40.914029 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":319,"threads_started":1}
I20260619 13:01:40.916486 12170 heartbeater.cc:347] Connected to a master server at 127.1.87.126:43621
I20260619 13:01:40.916735 12170 heartbeater.cc:464] Registering TS with master...
I20260619 13:01:40.917395 12170 heartbeater.cc:511] Master 127.1.87.126:43621 requested a full tablet report, sending...
I20260619 13:01:40.918618 11714 ts_manager.cc:194] Registered new tserver with Master: b36bbf39e37c4c4cab95e48f48ff1fb1 (127.1.87.67:39403)
I20260619 13:01:40.919497 11714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:50303
I20260619 13:01:40.926476 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:01:40.942035 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.930832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.1.87.126:43621)
0619 13:01:40.930977 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:40.930981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:40.931533 (+ 552us) server_negotiation.cc:407] Connection header received
0619 13:01:40.931654 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:40.931658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:40.931721 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:40.931804 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:40.933047 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.933735 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.934900 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.935289 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.935440 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.937437 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:40.937441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:40.937444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:40.937472 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:40.939816 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:40.940375 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:40.940378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:40.940381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:40.940497 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:40.940816 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:40.940818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:40.940820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:40.941095 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:40.941285 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:40.941523 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 13:01:40.941672 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:01:40.945976 11714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:01:40.952560 11714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:01:40.986869 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.963967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:39403 (local address 127.0.0.1:54808)
0619 13:01:40.970568 (+ 6601us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:40.970594 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:40.970756 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:40.979185 (+ 8429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:40.979195 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:40.979228 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:40.979805 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.979819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.981482 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.981490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.983885 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.983900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.985738 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.985744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.985943 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.986011 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 13:01:40.986159 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 13:01:40.986361 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6431,"mutex_wait_us":29,"thread_start_us":1023,"threads_started":1}
I20260619 13:01:40.987231 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.965888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:41865 (local address 127.0.0.1:42206)
0619 13:01:40.978243 (+ 12355us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:40.978264 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:40.978395 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:40.979596 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:40.979603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:40.979630 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:40.980197 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.980209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.982129 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.982135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.984527 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.984544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.986202 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.986208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.986353 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.986379 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 13:01:40.986451 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 13:01:40.986595 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12198,"mutex_wait_us":43,"thread_start_us":13053,"threads_started":1}
I20260619 13:01:40.987795 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.963834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:41255 (local address 127.0.0.1:48386)
0619 13:01:40.965421 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:40.965471 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:40.965666 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:40.979547 (+ 13881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:40.979551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:40.979577 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:40.980093 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.980110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.982155 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.982159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.984995 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:40.985010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.986898 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.986905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:40.987093 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.987128 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:01:40.987194 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 13:01:40.987378 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"mutex_wait_us":178,"thread_start_us":384,"threads_started":1}
I20260619 13:01:40.988507 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.964530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.1.87.67:39403)
0619 13:01:40.978449 (+ 13919us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:40.978459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:40.978493 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:01:40.978624 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:40.978632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:40.978921 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:40.980032 (+ 1111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:40.980296 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.981349 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.984049 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.985531 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.985692 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.985943 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:40.986427 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 13:01:40.986588 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13729,"thread_start_us":13608,"threads_started":1}
I20260619 13:01:40.988617 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.963915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.1.87.65:41255)
0619 13:01:40.978888 (+ 14973us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:40.978896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:40.978926 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:01:40.979061 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:40.979070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:40.979305 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:40.980243 (+ 938us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:40.980512 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.981999 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.985196 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.986644 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.987777 (+ 1133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.988013 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:40.988091 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 13:01:40.988261 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14826,"thread_start_us":14665,"threads_started":1}
I20260619 13:01:40.988792 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:40.966171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42206 (local address 127.1.87.66:41865)
0619 13:01:40.978819 (+ 12648us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:40.978826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:40.978851 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:01:40.978939 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:40.978948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:40.979163 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:40.979366 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:40.980347 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.981978 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.984708 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:40.985979 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:40.986125 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:40.986299 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:40.987489 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0619 13:01:40.988404 (+ 915us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12501,"thread_start_us":12589,"threads_started":1}
I20260619 13:01:40.992370 11972 tablet_service.cc:1467] Processing CreateTablet for tablet ce9ea26db1a046a19375287ab2d6e706 (DEFAULT_TABLE table=test-table [id=8ea6cf5759b04b9b99e5f3dfa00d4f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:40.992369 12105 tablet_service.cc:1467] Processing CreateTablet for tablet ce9ea26db1a046a19375287ab2d6e706 (DEFAULT_TABLE table=test-table [id=8ea6cf5759b04b9b99e5f3dfa00d4f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:40.992369 11838 tablet_service.cc:1467] Processing CreateTablet for tablet ce9ea26db1a046a19375287ab2d6e706 (DEFAULT_TABLE table=test-table [id=8ea6cf5759b04b9b99e5f3dfa00d4f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:01:40.993628 11972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9ea26db1a046a19375287ab2d6e706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:40.993682 12105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9ea26db1a046a19375287ab2d6e706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:40.993700 11838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9ea26db1a046a19375287ab2d6e706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:01:41.005785 12188 tablet_bootstrap.cc:492] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: Bootstrap starting.
I20260619 13:01:41.008330 12190 tablet_bootstrap.cc:492] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: Bootstrap starting.
I20260619 13:01:41.010260 12188 tablet_bootstrap.cc:654] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:41.010246 12189 tablet_bootstrap.cc:492] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: Bootstrap starting.
I20260619 13:01:41.011628 12188 log.cc:824] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:41.013367 12190 tablet_bootstrap.cc:654] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:41.013471 12189 tablet_bootstrap.cc:654] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: Neither blocks nor log segments found. Creating new log.
I20260619 13:01:41.014518 12190 log.cc:824] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:41.014657 12189 log.cc:824] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: Log is configured to *not* fsync() on all Append() calls
I20260619 13:01:41.014918 12188 tablet_bootstrap.cc:492] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: No bootstrap required, opened a new log
I20260619 13:01:41.015210 12188 ts_tablet_manager.cc:1397] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 13:01:41.017659 12189 tablet_bootstrap.cc:492] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: No bootstrap required, opened a new log
I20260619 13:01:41.017818 12190 tablet_bootstrap.cc:492] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: No bootstrap required, opened a new log
I20260619 13:01:41.018074 12190 ts_tablet_manager.cc:1397] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260619 13:01:41.018075 12189 ts_tablet_manager.cc:1397] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260619 13:01:41.025862 12189 raft_consensus.cc:348] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } }
I20260619 13:01:41.026271 12189 raft_consensus.cc:374] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:41.025974 12188 raft_consensus.cc:348] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } }
I20260619 13:01:41.026361 12189 raft_consensus.cc:729] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6faa4ca72ba64e91a82c098000f103fc, State: Initialized, Role: FOLLOWER
I20260619 13:01:41.026399 12188 raft_consensus.cc:374] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:41.026494 12188 raft_consensus.cc:729] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc300b137af4ae0a7a7a1fa9c067850, State: Initialized, Role: FOLLOWER
I20260619 13:01:41.027082 12189 consensus_queue.cc:260] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } }
I20260619 13:01:41.027117 12188 consensus_queue.cc:260] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } }
I20260619 13:01:41.029124 12189 ts_tablet_manager.cc:1428] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260619 13:01:41.028955 12190 raft_consensus.cc:348] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } }
I20260619 13:01:41.029402 12190 raft_consensus.cc:374] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:01:41.029507 12190 raft_consensus.cc:729] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36bbf39e37c4c4cab95e48f48ff1fb1, State: Initialized, Role: FOLLOWER
I20260619 13:01:41.030265 12190 consensus_queue.cc:260] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } }
I20260619 13:01:41.031535 12170 heartbeater.cc:503] Master 127.1.87.126:43621 was elected leader, sending a full tablet report...
I20260619 13:01:41.032418 12190 ts_tablet_manager.cc:1428] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260619 13:01:41.033346 12037 heartbeater.cc:503] Master 127.1.87.126:43621 was elected leader, sending a full tablet report...
I20260619 13:01:41.034330 12188 ts_tablet_manager.cc:1428] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: Time spent starting tablet: real 0.019s user 0.013s sys 0.003s
W20260619 13:01:41.121107 11904 tablet.cc:2369] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:01:41.129459 12038 tablet.cc:2369] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:01:41.144761 12171 tablet.cc:2369] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:01:41.231055 12196 raft_consensus.cc:482] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:01:41.231415 12196 raft_consensus.cc:504] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } }
I20260619 13:01:41.233114 12196 leader_election.cc:288] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6faa4ca72ba64e91a82c098000f103fc (127.1.87.65:41255), edc300b137af4ae0a7a7a1fa9c067850 (127.1.87.66:41865)
I20260619 13:01:41.238843 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:41.233294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:41255 (local address 127.1.87.67:34397)
0619 13:01:41.233654 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:41.233678 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:41.233816 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:41.234254 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:41.234265 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:41.234302 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:41.234499 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:41.234508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.235854 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.235860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:41.237075 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:41.237089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.238145 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.238151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:41.238324 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:41.238347 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:01:41.238451 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 13:01:41.238576 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260619 13:01:41.238843 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:41.233549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:34397 (local address 127.1.87.65:41255)
0619 13:01:41.233788 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:41.233797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:41.233822 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:01:41.234001 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:41.234007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:41.234092 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:41.234211 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:41.234673 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.235729 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.237226 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.237993 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.238145 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:41.238235 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:41.238475 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 13:01:41.238592 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:01:41.239717 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:41.233686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:57619 (local address 127.1.87.66:41865)
0619 13:01:41.234055 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:41.234064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:41.234577 (+ 513us) server_negotiation.cc:407] Connection header received
0619 13:01:41.234742 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:41.234745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:41.234802 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:41.234871 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:01:41.235603 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.236362 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.237995 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.238745 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.238899 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:41.238993 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:41.239341 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 13:01:41.239465 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260619 13:01:41.239750 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:41.233608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:41865 (local address 127.1.87.67:57619)
0619 13:01:41.234478 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 13:01:41.234494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:01:41.234585 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:01:41.234920 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:01:41.234924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:01:41.234951 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:01:41.235271 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:41.235283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.236515 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.236521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:41.237837 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:01:41.237848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.238971 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.238977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:01:41.239152 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:41.239182 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:01:41.239277 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 13:01:41.239433 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":568,"threads_started":1}
I20260619 13:01:41.239892 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9ea26db1a046a19375287ab2d6e706" candidate_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6faa4ca72ba64e91a82c098000f103fc" is_pre_election: true
I20260619 13:01:41.240500 11858 raft_consensus.cc:2454] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b36bbf39e37c4c4cab95e48f48ff1fb1 in term 0.
I20260619 13:01:41.240751 11992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9ea26db1a046a19375287ab2d6e706" candidate_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc300b137af4ae0a7a7a1fa9c067850" is_pre_election: true
I20260619 13:01:41.241127 12058 leader_election.cc:302] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6faa4ca72ba64e91a82c098000f103fc, b36bbf39e37c4c4cab95e48f48ff1fb1; no voters:
I20260619 13:01:41.241178 11992 raft_consensus.cc:2454] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b36bbf39e37c4c4cab95e48f48ff1fb1 in term 0.
I20260619 13:01:41.241490 12196 raft_consensus.cc:2781] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:01:41.241657 12196 raft_consensus.cc:482] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:01:41.241796 12196 raft_consensus.cc:3037] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:41.244026 12196 raft_consensus.cc:504] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } }
I20260619 13:01:41.244596 12196 leader_election.cc:288] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [CANDIDATE]: Term 1 election: Requested vote from peers 6faa4ca72ba64e91a82c098000f103fc (127.1.87.65:41255), edc300b137af4ae0a7a7a1fa9c067850 (127.1.87.66:41865)
I20260619 13:01:41.244990 11992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9ea26db1a046a19375287ab2d6e706" candidate_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc300b137af4ae0a7a7a1fa9c067850"
I20260619 13:01:41.244992 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9ea26db1a046a19375287ab2d6e706" candidate_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6faa4ca72ba64e91a82c098000f103fc"
I20260619 13:01:41.245163 11992 raft_consensus.cc:3037] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:41.245174 11858 raft_consensus.cc:3037] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:01:41.247184 11858 raft_consensus.cc:2454] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b36bbf39e37c4c4cab95e48f48ff1fb1 in term 1.
I20260619 13:01:41.247339 11992 raft_consensus.cc:2454] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b36bbf39e37c4c4cab95e48f48ff1fb1 in term 1.
I20260619 13:01:41.247696 12059 leader_election.cc:302] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36bbf39e37c4c4cab95e48f48ff1fb1, edc300b137af4ae0a7a7a1fa9c067850; no voters:
I20260619 13:01:41.248021 12196 raft_consensus.cc:2781] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:01:41.248464 12196 raft_consensus.cc:686] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 1 LEADER]: Becoming Leader. State: Replica: b36bbf39e37c4c4cab95e48f48ff1fb1, State: Running, Role: LEADER
I20260619 13:01:41.248950 12196 consensus_queue.cc:237] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } }
I20260619 13:01:41.253139 11714 catalog_manager.cc:5495] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b36bbf39e37c4c4cab95e48f48ff1fb1 (127.1.87.67). New cstate: current_term: 1 leader_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b36bbf39e37c4c4cab95e48f48ff1fb1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 } health_report { overall_health: UNKNOWN } } }
I20260619 13:01:41.285435 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:41.280273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.1.87.67:39403)
0619 13:01:41.280467 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:41.280475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:41.280584 (+ 109us) server_negotiation.cc:407] Connection header received
0619 13:01:41.280831 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:41.280835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:41.280910 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:41.281073 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:01:41.281414 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.282114 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.283414 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.283815 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.283948 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:41.284419 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:01:41.284883 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:01:41.285046 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:41.285100 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 13:01:41.285174 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:01:41.292846 11992 raft_consensus.cc:1261] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [term 1 FOLLOWER]: Refusing update from remote peer b36bbf39e37c4c4cab95e48f48ff1fb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:41.292862 11858 raft_consensus.cc:1261] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [term 1 FOLLOWER]: Refusing update from remote peer b36bbf39e37c4c4cab95e48f48ff1fb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:01:41.293784 12198 consensus_queue.cc:1035] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6faa4ca72ba64e91a82c098000f103fc" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:41.294106 12196 consensus_queue.cc:1035] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc300b137af4ae0a7a7a1fa9c067850" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:01:41.305199 12201 mvcc.cc:205] Tried to move back new op lower bound from 7298556318886621184 to 7298556318719172608. Current Snapshot: MvccSnapshot[applied={T|T < 7298556318886621184}]
I20260619 13:01:41.306210 12203 mvcc.cc:205] Tried to move back new op lower bound from 7298556318886621184 to 7298556318719172608. Current Snapshot: MvccSnapshot[applied={T|T < 7298556318886621184}]
I20260619 13:01:41.310935 12202 mvcc.cc:205] Tried to move back new op lower bound from 7298556318886621184 to 7298556318719172608. Current Snapshot: MvccSnapshot[applied={T|T < 7298556318886621184 or (T in {7298556318886621184})}]
I20260619 13:01:41.317790 11714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:41.318017 11714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:01:41.321309 11714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8138562ba164c6d9164ad35d5f5b8dc: Sending DeleteTablet for 3 replicas of tablet ce9ea26db1a046a19375287ab2d6e706
I20260619 13:01:41.322391 11838 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9ea26db1a046a19375287ab2d6e706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:41 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:48386
I20260619 13:01:41.322394 12105 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9ea26db1a046a19375287ab2d6e706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:41 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:54808
I20260619 13:01:41.322876 11972 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9ea26db1a046a19375287ab2d6e706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:01:41 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:42206
I20260619 13:01:41.324793 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 11775
I20260619 13:01:41.334082 12212 tablet_replica.cc:330] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: stopping tablet replica
I20260619 13:01:41.335330 12212 raft_consensus.cc:2229] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:41.336622 12212 raft_consensus.cc:2258] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:41.337488 12210 tablet_replica.cc:330] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: stopping tablet replica
I20260619 13:01:41.338392 12210 raft_consensus.cc:2229] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:01:41.339001 12210 raft_consensus.cc:2258] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:41.339844 12212 ts_tablet_manager.cc:1905] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:41.341012 12210 ts_tablet_manager.cc:1905] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:41.347970 12210 ts_tablet_manager.cc:1918] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:41.348143 12210 log.cc:1196] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/ce9ea26db1a046a19375287ab2d6e706
I20260619 13:01:41.348531 12211 tablet_replica.cc:330] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: stopping tablet replica
I20260619 13:01:41.348876 12211 raft_consensus.cc:2229] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:01:41.349359 12211 raft_consensus.cc:2258] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:01:41.350025 12210 ts_tablet_manager.cc:1939] T ce9ea26db1a046a19375287ab2d6e706 P b36bbf39e37c4c4cab95e48f48ff1fb1: Deleting consensus metadata
I20260619 13:01:41.350545 12212 ts_tablet_manager.cc:1918] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:41.350730 12212 log.cc:1196] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/ce9ea26db1a046a19375287ab2d6e706
I20260619 13:01:41.351229 12212 ts_tablet_manager.cc:1939] T ce9ea26db1a046a19375287ab2d6e706 P edc300b137af4ae0a7a7a1fa9c067850: Deleting consensus metadata
I20260619 13:01:41.351796 12211 ts_tablet_manager.cc:1905] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:01:41.352109 11700 catalog_manager.cc:4841] TS b36bbf39e37c4c4cab95e48f48ff1fb1 (127.1.87.67:39403): tablet ce9ea26db1a046a19375287ab2d6e706 (table test-table [id=8ea6cf5759b04b9b99e5f3dfa00d4f94]) successfully deleted
I20260619 13:01:41.353231 11699 catalog_manager.cc:4841] TS edc300b137af4ae0a7a7a1fa9c067850 (127.1.87.66:41865): tablet ce9ea26db1a046a19375287ab2d6e706 (table test-table [id=8ea6cf5759b04b9b99e5f3dfa00d4f94]) successfully deleted
Jun 19 13:01:41 dist-test-slave-tpsm krb5kdc[11669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:01:41.358489 12211 ts_tablet_manager.cc:1918] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:01:41.358654 12211 log.cc:1196] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/ce9ea26db1a046a19375287ab2d6e706
I20260619 13:01:41.359170 12211 ts_tablet_manager.cc:1939] T ce9ea26db1a046a19375287ab2d6e706 P 6faa4ca72ba64e91a82c098000f103fc: Deleting consensus metadata
I20260619 13:01:41.360922 11698 catalog_manager.cc:4841] TS 6faa4ca72ba64e91a82c098000f103fc (127.1.87.65:41255): tablet ce9ea26db1a046a19375287ab2d6e706 (table test-table [id=8ea6cf5759b04b9b99e5f3dfa00d4f94]) successfully deleted
I20260619 13:01:41.364693 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:41.326340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48402 (local address 127.1.87.65:41255)
0619 13:01:41.326553 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:41.326562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:41.337224 (+ 10662us) server_negotiation.cc:407] Connection header received
0619 13:01:41.337681 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:41.337686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:41.337773 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:41.337865 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:41.341566 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.342695 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.348232 (+ 5537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.348742 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.348820 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:41.358964 (+ 10144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:41.358978 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:41.358992 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:41.359025 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:41.361980 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:41.362637 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:41.362644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:41.362646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:41.362766 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:41.363147 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:41.363156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:41.363158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:41.363568 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:41.363865 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:41.364146 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 13:01:41.364348 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:01:41.365306 11898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:41.468278 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 11775
I20260619 13:01:41.486522 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 11909
Jun 19 13:01:41 dist-test-slave-tpsm krb5kdc[11669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:01:41.503460 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:41.487448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.1.87.66:41865)
0619 13:01:41.487642 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:41.487650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:41.487671 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:01:41.487830 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:41.487836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:41.487924 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:41.488047 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:41.489425 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.490589 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.491937 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.492603 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.492744 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:41.496712 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:41.496730 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:41.496749 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:41.496793 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:41.500597 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:41.501367 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:41.501377 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:41.501381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:41.501528 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:41.501875 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:41.501880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:41.501884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:41.502360 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:41.502644 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:41.502846 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 13:01:41.503058 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:01:41.504287 12032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:41.594332 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 11909
I20260619 13:01:41.611655 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 12042
Jun 19 13:01:41 dist-test-slave-tpsm krb5kdc[11669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:01:41.628032 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:01:41.612562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.1.87.67:39403)
0619 13:01:41.612746 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:01:41.612752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:01:41.612780 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:01:41.612961 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:01:41.612966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:01:41.613051 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:01:41.613160 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:01:41.614508 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.615403 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.616778 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:01:41.617234 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:01:41.617382 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:01:41.621892 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:01:41.621916 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:01:41.621937 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:01:41.621982 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:01:41.625218 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:41.625944 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:41.625951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:41.625953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:41.626098 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:01:41.626478 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:01:41.626484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:01:41.626487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:01:41.626831 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:01:41.627161 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 13:01:41.627416 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 13:01:41.627643 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:01:41.628731 12165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:41.749653 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 12042
I20260619 13:01:41.769449 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 11682
I20260619 13:01:41.770553 11744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:01:41.879441 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 11682
2026-06-19T13:01:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4381 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:01:41 dist-test-slave-tpsm krb5kdc[12220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:01:41 dist-test-slave-tpsm krb5kdc[12220](info): set up 2 sockets
Jun 19 13:01:41 dist-test-slave-tpsm krb5kdc[12220](info): commencing operation
krb5kdc: starting...
W20260619 13:01:42.980335 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:01:43 dist-test-slave-tpsm krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:01:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.87.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.87.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/rangeradmin_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/rangeradmin_127.1.87.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.87.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.87.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/rangerlookup_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/rangerlookup_127.1.87.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.87.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.87.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/HTTP_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/HTTP_127.1.87.84@KRBTEST.COM.keytab.
I20260619 13:01:43.291280 1373 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskas476m/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/postgres -l logfile start
2026-06-19 13:01:45.578 UTC [12250] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 13:01:45.578 UTC [12250] LOG: listening on IPv4 address "127.1.87.84", port 48979
2026-06-19 13:01:45.581 UTC [12250] LOG: listening on Unix socket "/tmp/.s.PGSQL.48979"
2026-06-19 13:01:45.603 UTC [12253] LOG: database system was shut down at 2026-06-19 13:01:44 UTC
2026-06-19 13:01:45.607 UTC [12250] LOG: database system is ready to accept connections
I20260619 13:01:46.579986 1373 mini_postgres.cc:96] Postgres bound to 48979
2026-06-19 13:01:46.597 UTC [12261] FATAL: database "slave" does not exist
127.1.87.84:48979 - accepting connections
I20260619 13:01:46.598358 1373 mini_ranger.cc:161] Starting Ranger...
I20260619 13:01:46.625959 1373 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 13:01:47.081194 1373 mini_ranger.cc:87] Created ranger Postgres database
I20260619 13:01:47.081307 1373 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-admin
2026-06-19 13:01:47,402 [I] DB FLAVOR :POSTGRES
2026-06-19 13:01:47,403 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:01:47,403 [I] Checking connection..
2026-06-19 13:01:47,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 13:01:47,645 [I] Checking connection passed.
2026-06-19 13:01:47,646 [I] --------- Verifying version history table ---------
2026-06-19 13:01:47,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:01:47,904 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:01:47,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:01:48,134 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:01:48,134 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 13:01:48,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 13:01:48.383 UTC [12325] WARNING: there is no transaction in progress
2026-06-19 13:01:48,390 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 13:01:48,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:01:48,652 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 13:01:48,653 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:01:48,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 13:01:48,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 13:01:49,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tpsm.c.gcp-upstream.internal','N') ;"
2026-06-19 13:01:49,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:01:49,625 [I] Table x_portal_user does not exist in database ranger
2026-06-19 13:01:49,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:01:49,871 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 13:01:49,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:01:50,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:01:50,373 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 13:01:50,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 13:01:51.271 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.286 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.300 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.314 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.329 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.346 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.408 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.414 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.427 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.436 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.445 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.458 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.468 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.474 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.480 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:51.485 UTC [12433] WARNING: there is no transaction in progress
2026-06-19 13:01:52,050 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 13:01:52,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 13:01:52,268 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 13:01:52,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:01:52,524 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 13:01:52,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:01:52,794 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 13:01:52,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:01:53,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:01:53,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tpsm.c.gcp-upstream.internal';"
2026-06-19 13:01:53,523 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 13:01:53,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 13:01:53,754 [I] DB_PATCHES have already been applied
I20260619 13:01:53.760807 1373 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-admin:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskas476m/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 13:01:53.760900 1373 mini_ranger.cc:193] Using host: 127.1.87.84
I20260619 13:01:53.775810 1373 mini_ranger.cc:227] Ranger admin URL: http://127.1.87.84:40507
Jun 19, 2026 1:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 13:01:54.892967 1373 mini_ranger.cc:160] Time spent starting Ranger: real 8.295s user 0.000s sys 0.042s
Jun 19, 2026 1:02:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:02:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 1:02:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:02:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:02:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 13:02:17.233989 1373 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.87.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.87.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/rangerkms_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/rangerkms_127.1.87.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.87.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.87.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.87.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/HTTP_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.87.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/HTTP_127.1.87.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 13:02:17 dist-test-slave-tpsm krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874137, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 13:02:17.408747 1373 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 13:02:17.433241 1373 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 13:02:18.199651 1373 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 13:02:18.199813 1373 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-kms
I20260619 13:02:18.199965 1373 mini_ranger_kms.cc:227] Using postgres at 127.1.87.84:48979
2026-06-19 13:02:18,622 [I] DB FLAVOR :POSTGRES
2026-06-19 13:02:18,623 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:02:18,623 [I] Checking connection
2026-06-19 13:02:18,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 13:02:18,874 [I] connection success
2026-06-19 13:02:18,874 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 13:02:18,874 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 13:02:18,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 13:02:19,128 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 13:02:19,128 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:02:19,129 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 13:02:19,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:48979/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 13:02:19,395 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 13:02:19.867712 1373 mini_ranger_kms.cc:310] Created kms service
I20260619 13:02:20.046552 1373 mini_ranger_kms.cc:326] Created kudu user
I20260619 13:02:20.088239 1373 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 13:02:20 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874137, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:02:20.469667 1373 mini_ranger_kms.cc:384] Added ranger policy
I20260619 13:02:20.470009 1373 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-kms:/tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 13:02:20.470062 1373 mini_ranger_kms.cc:240] Using host: 127.1.87.84
I20260619 13:02:20.489379 1373 mini_ranger_kms.cc:276] Ranger KMS PID: 12613
I20260619 13:02:20.489576 1373 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.87.84:51075
Jun 19, 2026 1:02:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:02:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:02:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:02:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:02:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:02:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 13:02:22 dist-test-slave-tpsm krb5kdc[12220](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874142, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.87.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:02:22.803750 1373 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.395s user 0.081s sys 0.365s
I20260619 13:02:22.804036 1373 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 13:02:22.804273 1373 mini_ranger_kms.cc:401] 127.1.87.84:51075/kms/v1/keys
Jun 19 13:02:23 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874142, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.87.84@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
Jun 19 13:02:23 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874142, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.87.84@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
Jun 19 13:02:25 dist-test-slave-tpsm krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:02:25.744410 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:40409
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:40409
--ranger_config_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:51075/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:02:26.089471 12664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:02:26.089803 12664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:02:26.089951 12664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:02:26.097224 12664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:02:26.097375 12664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:02:26.097476 12664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:02:26.097571 12664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:02:26.097654 12664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:02:26.107609 12664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.87.84:51075/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:40409
--ranger_config_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:40409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:02:26.109318 12664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:02:26.110627 12664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:02:26.122730 12670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:02:26.122702 12669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:02:26.122766 12672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:02:26.123450 12664 server_base.cc:1034] running on GCE node
I20260619 13:02:26.124572 12664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:02:26.126775 12664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:02:26.128010 12664 hybrid_clock.cc:648] HybridClock initialized: now 1781874146127944 us; error 58 us; skew 500 ppm
I20260619 13:02:26.128438 12664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:02:26 dist-test-slave-tpsm krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:02:26.131916 12664 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:02:26.133661 12664 webserver.cc:460] Webserver started at http://127.1.87.126:46443/ using document root <none> and password file <none>
I20260619 13:02:26.134333 12664 fs_manager.cc:359] Metadata directory not provided
I20260619 13:02:26.134462 12664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:02:26.134780 12664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:02:26 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:02:26.186703 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "9adedf9dcf1a410db873dc182d8aed53"
format_stamp: "Formatted at 2026-06-19 13:02:26 on dist-test-slave-tpsm"
server_key: "740adedaad61d838957f7540dd1a45c3"
server_key_iv: "512843f2331e9ae58fda51a60d1409ba"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:26.188031 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "9adedf9dcf1a410db873dc182d8aed53"
format_stamp: "Formatted at 2026-06-19 13:02:26 on dist-test-slave-tpsm"
server_key: "740adedaad61d838957f7540dd1a45c3"
server_key_iv: "512843f2331e9ae58fda51a60d1409ba"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:26.193250 12664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:02:26.212347 12679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:26.213877 12664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:02:26.214056 12664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "9adedf9dcf1a410db873dc182d8aed53"
format_stamp: "Formatted at 2026-06-19 13:02:26 on dist-test-slave-tpsm"
server_key: "740adedaad61d838957f7540dd1a45c3"
server_key_iv: "512843f2331e9ae58fda51a60d1409ba"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:26.214249 12664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:02:26.228739 12664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:02:26.233642 12664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:02:26.233896 12664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:02:26.255008 12664 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:40409
I20260619 13:02:26.255040 12730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:40409 every 8 connection(s)
I20260619 13:02:26.256345 12664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:02:26.260715 12731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:02:26.265182 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 12664
I20260619 13:02:26.265400 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:02:26.266023 1373 external_mini_cluster.cc:1428] Setting key 5e20f4f0874bf212bf555f6af7306fe9
I20260619 13:02:26.272624 12731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53: Bootstrap starting.
I20260619 13:02:26.276609 12731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53: Neither blocks nor log segments found. Creating new log.
I20260619 13:02:26.277714 12731 log.cc:824] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53: Log is configured to *not* fsync() on all Append() calls
I20260619 13:02:26.280830 12731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53: No bootstrap required, opened a new log
Jun 19 13:02:26 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:02:26.289378 12731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9adedf9dcf1a410db873dc182d8aed53" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 40409 } }
I20260619 13:02:26.289777 12731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:02:26.289896 12731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9adedf9dcf1a410db873dc182d8aed53, State: Initialized, Role: FOLLOWER
I20260619 13:02:26.290191 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:26.268045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.1.87.126:40409)
0619 13:02:26.268866 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:26.268880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:26.268936 (+ 56us) server_negotiation.cc:407] Connection header received
0619 13:02:26.271387 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:26.271414 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:26.271902 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:26.272314 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:02:26.273493 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:26.275956 (+ 2463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:26.277255 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:26.278317 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:26.278491 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:26.282634 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:02:26.282641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:02:26.282658 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:02:26.282697 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:02:26.285847 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:26.286583 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:26.286601 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:26.286607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:26.286755 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:26.287133 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:26.287139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:26.287144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:26.287771 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:02:26.288284 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:26.288577 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 13:02:26.288942 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":342,"threads_started":1}
I20260619 13:02:26.290649 12731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9adedf9dcf1a410db873dc182d8aed53" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 40409 } }
I20260619 13:02:26.290946 12731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:02:26.291090 12731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:02:26.291278 12731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:02:26.293817 12731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9adedf9dcf1a410db873dc182d8aed53" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 40409 } }
I20260619 13:02:26.294303 12731 leader_election.cc:302] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9adedf9dcf1a410db873dc182d8aed53; no voters:
I20260619 13:02:26.294847 12731 leader_election.cc:288] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:02:26.294992 12736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:02:26.295692 12736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [term 1 LEADER]: Becoming Leader. State: Replica: 9adedf9dcf1a410db873dc182d8aed53, State: Running, Role: LEADER
I20260619 13:02:26.296218 12736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9adedf9dcf1a410db873dc182d8aed53" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 40409 } }
I20260619 13:02:26.297001 12731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:02:26.298974 12737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9adedf9dcf1a410db873dc182d8aed53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9adedf9dcf1a410db873dc182d8aed53" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 40409 } } }
I20260619 13:02:26.299639 12738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9adedf9dcf1a410db873dc182d8aed53. Latest consensus state: current_term: 1 leader_uuid: "9adedf9dcf1a410db873dc182d8aed53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9adedf9dcf1a410db873dc182d8aed53" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 40409 } } }
I20260619 13:02:26.300244 12737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [sys.catalog]: This master's current role is: LEADER
I20260619 13:02:26.300251 12738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53 [sys.catalog]: This master's current role is: LEADER
I20260619 13:02:26.301846 12743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:02:26.307595 12743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:02:26.307801 12731 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 13:02:26.325012 12743 catalog_manager.cc:1347] Generated new cluster ID: 01f42944ef764e6089330705d450146f
I20260619 13:02:26.325261 12743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:02:26.390094 12743 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:02:26.391644 12743 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:02:26.406311 12743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53: Generated new TSK 0
I20260619 13:02:26.407183 12743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:02:27.180820 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:51075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:40409
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:02:27 dist-test-slave-tpsm krb5kdc[12220](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874147, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 13:02:27.594699 12776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:02:27.595007 12776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:02:27.595186 12776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:02:27.604104 12776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:02:27.604301 12776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:02:27.604496 12776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:02:27.615908 12776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.87.84:51075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:40409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:02:27.617309 12776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:02:27.618765 12776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:02:27.637538 12786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:02:27.645026 12787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:02:27.645905 12789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:02:27.646446 12776 server_base.cc:1034] running on GCE node
I20260619 13:02:27.647138 12776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:02:27.648355 12776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:02:27.649642 12776 hybrid_clock.cc:648] HybridClock initialized: now 1781874147649552 us; error 82 us; skew 500 ppm
I20260619 13:02:27.649977 12776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:02:27 dist-test-slave-tpsm krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874147, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:02:27.653204 12776 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:02:27.656179 12776 webserver.cc:460] Webserver started at http://127.1.87.65:33493/ using document root <none> and password file <none>
I20260619 13:02:27.657027 12776 fs_manager.cc:359] Metadata directory not provided
I20260619 13:02:27.657196 12776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:02:27.657545 12776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:02:27 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874147, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:02:27.680758 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "c0cf8160326642be91bd34c57589f089"
format_stamp: "Formatted at 2026-06-19 13:02:27 on dist-test-slave-tpsm"
server_key: "ac56afd7a8b3864461af0020a9c1097b"
server_key_iv: "7e1e998b1ad0a4169d829f9023efb36c"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:27.681934 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "c0cf8160326642be91bd34c57589f089"
format_stamp: "Formatted at 2026-06-19 13:02:27 on dist-test-slave-tpsm"
server_key: "ac56afd7a8b3864461af0020a9c1097b"
server_key_iv: "7e1e998b1ad0a4169d829f9023efb36c"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:27.710444 12776 fs_manager.cc:691] Time spent creating directory manager: real 0.028s user 0.007s sys 0.019s
I20260619 13:02:27.732353 12796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:27.734225 12776 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:02:27.734501 12776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "c0cf8160326642be91bd34c57589f089"
format_stamp: "Formatted at 2026-06-19 13:02:27 on dist-test-slave-tpsm"
server_key: "ac56afd7a8b3864461af0020a9c1097b"
server_key_iv: "7e1e998b1ad0a4169d829f9023efb36c"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:27.734892 12776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:02:27.779807 12776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:02:27.784629 12776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:02:27.784953 12776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:02:27.786011 12776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:02:27.787601 12776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:02:27.787708 12776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:27.787855 12776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:02:27.787928 12776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:27.859596 12776 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:35743
I20260619 13:02:27.859596 12908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:35743 every 8 connection(s)
I20260619 13:02:27.861020 12776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:02:27.861720 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 12776
I20260619 13:02:27.861904 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:02:27.862702 1373 external_mini_cluster.cc:1428] Setting key 867c85fd8299ac6e4b852a0a83eb2351
Jun 19 13:02:27 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874147, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
I20260619 13:02:27.892174 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:27.867029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:45433 (local address 127.1.87.126:40409)
0619 13:02:27.872282 (+ 5253us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:27.872288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:27.872308 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:02:27.872367 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:27.872373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:27.872448 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:27.872559 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:02:27.874657 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:27.875873 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:27.878339 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:27.878999 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:27.879124 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:27.883806 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:02:27.883814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:02:27.883819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:02:27.883859 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:02:27.887984 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:27.888995 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:27.889001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:27.889004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:27.889154 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:27.889787 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:27.889792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:27.889796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:27.890159 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:02:27.890292 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:27.891484 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0619 13:02:27.891787 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5129,"thread_start_us":7220,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
I20260619 13:02:27.895114 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:27.865250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:40409 (local address 127.1.87.65:45433)
0619 13:02:27.866775 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0619 13:02:27.866828 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:02:27.869152 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:02:27.873003 (+ 3851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:02:27.873015 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:02:27.873637 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:02:27.874417 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:27.874442 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:27.876081 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:27.876087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:27.878155 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:27.878171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:27.879458 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:27.879466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:27.879755 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:27.880581 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:02:27.880611 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:02:27.883489 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:02:27.888174 (+ 4685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:02:27.888183 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:02:27.888202 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:02:27.888840 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:02:27.889328 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:02:27.889333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:02:27.889337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:02:27.889625 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:02:27.890380 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:02:27.890388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:02:27.890844 (+ 456us) client_negotiation.cc:769] Sending connection context
0619 13:02:27.891176 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 13:02:27.893058 (+ 1882us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"thread_start_us":988,"threads_started":1}
I20260619 13:02:27.900306 12909 heartbeater.cc:347] Connected to a master server at 127.1.87.126:40409
I20260619 13:02:27.900693 12909 heartbeater.cc:464] Registering TS with master...
I20260619 13:02:27.901611 12909 heartbeater.cc:511] Master 127.1.87.126:40409 requested a full tablet report, sending...
I20260619 13:02:27.904352 12696 ts_manager.cc:194] Registered new tserver with Master: c0cf8160326642be91bd34c57589f089 (127.1.87.65:35743)
I20260619 13:02:27.906432 12696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:45433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:02:27.935487 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:51075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:40409
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:02:28 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874147, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
Jun 19 13:02:28 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874147, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
W20260619 13:02:28.402496 12915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:02:28.402906 12915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:02:28.403069 12915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:02:28.416097 12915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:02:28.416282 12915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:02:28.416474 12915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:02:28.431214 12915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.87.84:51075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:40409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:02:28.433027 12915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:02:28.434654 12915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:02:28.492285 12922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:02:28.500623 12923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:02:28.510649 12926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:02:28.516599 12915 server_base.cc:1034] running on GCE node
I20260619 13:02:28.517593 12915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:02:28.524073 12915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:02:28.525404 12915 hybrid_clock.cc:648] HybridClock initialized: now 1781874148525295 us; error 112 us; skew 500 ppm
I20260619 13:02:28.525801 12915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:02:28 dist-test-slave-tpsm krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:02:28.532075 12915 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:02:28.563292 12915 webserver.cc:460] Webserver started at http://127.1.87.66:36887/ using document root <none> and password file <none>
I20260619 13:02:28.564256 12915 fs_manager.cc:359] Metadata directory not provided
I20260619 13:02:28.564427 12915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:02:28.564889 12915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:02:28 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:02:28.585230 12915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "d22f91eb2f3741a2abedda9f62fe1c11"
format_stamp: "Formatted at 2026-06-19 13:02:28 on dist-test-slave-tpsm"
server_key: "73017053518778c4fac73e478c150bbb"
server_key_iv: "fc13636d60a314d7295ae5edfd070ce5"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:28.586479 12915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "d22f91eb2f3741a2abedda9f62fe1c11"
format_stamp: "Formatted at 2026-06-19 13:02:28 on dist-test-slave-tpsm"
server_key: "73017053518778c4fac73e478c150bbb"
server_key_iv: "fc13636d60a314d7295ae5edfd070ce5"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:28.611099 12915 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.015s sys 0.007s
I20260619 13:02:28.649330 12932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:28.652585 12915 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.002s
I20260619 13:02:28.652896 12915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "d22f91eb2f3741a2abedda9f62fe1c11"
format_stamp: "Formatted at 2026-06-19 13:02:28 on dist-test-slave-tpsm"
server_key: "73017053518778c4fac73e478c150bbb"
server_key_iv: "fc13636d60a314d7295ae5edfd070ce5"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:28.653261 12915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:02:28.706512 12915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:02:28.712036 12915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:02:28.712456 12915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:02:28.713675 12915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:02:28.715497 12915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:02:28.715587 12915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:28.715703 12915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:02:28.715765 12915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:28.759655 12915 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:37295
I20260619 13:02:28.759701 13051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:37295 every 8 connection(s)
I20260619 13:02:28.761440 12915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:02:28.766448 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 12915
I20260619 13:02:28.766611 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:02:28.767112 1373 external_mini_cluster.cc:1428] Setting key 592b5a797bad52eed0ed146da63f2191
Jun 19 13:02:28 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
I20260619 13:02:28.811648 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:28.765480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:59103 (local address 127.1.87.126:40409)
0619 13:02:28.790087 (+ 24607us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:28.790094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:28.793818 (+ 3724us) server_negotiation.cc:407] Connection header received
0619 13:02:28.793893 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:28.793899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:28.793999 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:28.794107 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:02:28.797441 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:28.798429 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:28.800154 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:28.800540 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:28.800690 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:28.804227 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:02:28.804232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:02:28.804235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:02:28.804269 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:02:28.807680 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:28.808667 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:28.808675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:28.808681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:28.808825 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:28.809423 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:28.809431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:28.809437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:28.809830 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:02:28.809984 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:28.810879 (+ 895us) server_negotiation.cc:299] Negotiation successful
0619 13:02:28.811133 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24461,"thread_start_us":24629,"threads_started":1}
I20260619 13:02:28.812633 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:28.764880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:40409 (local address 127.1.87.66:59103)
0619 13:02:28.790761 (+ 25881us) negotiation.cc:107] Waiting for socket to connect
0619 13:02:28.790822 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 13:02:28.793329 (+ 2507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:02:28.795352 (+ 2023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:02:28.795369 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:02:28.796012 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:02:28.797224 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:28.797250 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:28.798598 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:28.798603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:28.799979 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:28.799992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:28.800798 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:28.800804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:28.801043 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:28.801829 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:02:28.801866 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:02:28.803989 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:02:28.807850 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:02:28.807860 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:02:28.807878 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:02:28.808508 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:02:28.808962 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:02:28.808967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:02:28.808970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:02:28.809280 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:02:28.809995 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:02:28.810003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:02:28.810426 (+ 423us) client_negotiation.cc:769] Sending connection context
0619 13:02:28.810728 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 13:02:28.811262 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25533,"thread_start_us":29239,"threads_started":1}
I20260619 13:02:28.814284 13052 heartbeater.cc:347] Connected to a master server at 127.1.87.126:40409
I20260619 13:02:28.814630 13052 heartbeater.cc:464] Registering TS with master...
I20260619 13:02:28.815467 13052 heartbeater.cc:511] Master 127.1.87.126:40409 requested a full tablet report, sending...
I20260619 13:02:28.816913 12696 ts_manager.cc:194] Registered new tserver with Master: d22f91eb2f3741a2abedda9f62fe1c11 (127.1.87.66:37295)
I20260619 13:02:28.817824 12696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:59103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:02:28.838663 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:51075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:40409
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:02:28.910269 12909 heartbeater.cc:503] Master 127.1.87.126:40409 was elected leader, sending a full tablet report...
W20260619 13:02:29.210192 13058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:02:29.210557 13058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:02:29.210747 13058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:02:29.218555 13058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:02:29.218725 13058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:02:29.218924 13058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:02:29.229341 13058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.87.84:51075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:40409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:02:29.230661 13058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:02:29.231961 13058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:02:29.245389 13066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:02:29.245434 13064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:02:29.245525 13063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:02:29.246021 13058 server_base.cc:1034] running on GCE node
I20260619 13:02:29.246693 13058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:02:29.247686 13058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:02:29.248957 13058 hybrid_clock.cc:648] HybridClock initialized: now 1781874149248925 us; error 51 us; skew 500 ppm
I20260619 13:02:29.249347 13058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:02:29 dist-test-slave-tpsm krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:02:29.252674 13058 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:02:29.254278 13058 webserver.cc:460] Webserver started at http://127.1.87.67:40261/ using document root <none> and password file <none>
I20260619 13:02:29.255496 13058 fs_manager.cc:359] Metadata directory not provided
I20260619 13:02:29.255635 13058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:02:29.255987 13058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:02:29 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:02:29.272364 13058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "89e5878dec0b4d1aacd41b187dce6547"
format_stamp: "Formatted at 2026-06-19 13:02:29 on dist-test-slave-tpsm"
server_key: "8fad77c62c448d4c4cbd6ff21a7ec099"
server_key_iv: "4981fa1731b027fe352eea99ed455a37"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:29.273262 13058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "89e5878dec0b4d1aacd41b187dce6547"
format_stamp: "Formatted at 2026-06-19 13:02:29 on dist-test-slave-tpsm"
server_key: "8fad77c62c448d4c4cbd6ff21a7ec099"
server_key_iv: "4981fa1731b027fe352eea99ed455a37"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:29.278652 13058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 13:02:29.296566 13073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:29.298343 13058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 13:02:29.298542 13058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "89e5878dec0b4d1aacd41b187dce6547"
format_stamp: "Formatted at 2026-06-19 13:02:29 on dist-test-slave-tpsm"
server_key: "8fad77c62c448d4c4cbd6ff21a7ec099"
server_key_iv: "4981fa1731b027fe352eea99ed455a37"
server_key_version: "kuduclusterkey@0"
I20260619 13:02:29.298799 13058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:02:29.334012 13058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:02:29.337831 13058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:02:29.338145 13058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:02:29.339102 13058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:02:29.340972 13058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:02:29.341051 13058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:29.341194 13058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:02:29.341254 13058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:02:29.376394 13058 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:42523
I20260619 13:02:29.376426 13185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:42523 every 8 connection(s)
I20260619 13:02:29.377635 13058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:02:29.378777 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 13058
I20260619 13:02:29.378947 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:02:29.379503 1373 external_mini_cluster.cc:1428] Setting key a5875dec066ea766669745d83054eab3
Jun 19 13:02:29 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:02:29.398206 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.380467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:40601 (local address 127.1.87.126:40409)
0619 13:02:29.380965 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:29.380971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:29.382051 (+ 1080us) server_negotiation.cc:407] Connection header received
0619 13:02:29.383944 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:29.383948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:29.384032 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:29.384126 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:02:29.385488 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.386229 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.388179 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.388625 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.388729 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.391939 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:02:29.391944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:02:29.391947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:02:29.391977 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:02:29.395151 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:29.396056 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:29.396061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:29.396063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:29.396185 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:29.396625 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:29.396629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:29.396631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:29.396949 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:02:29.397072 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:29.397703 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 13:02:29.397948 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":226,"threads_started":1}
I20260619 13:02:29.399065 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.381066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:40409 (local address 127.1.87.67:40601)
0619 13:02:29.381890 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0619 13:02:29.381941 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:02:29.383681 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:02:29.384337 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:02:29.384345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:02:29.384769 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:02:29.385317 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.385334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.386366 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.386370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.388013 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.388024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.388827 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.388832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.389017 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.389682 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:02:29.389703 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:02:29.391741 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:02:29.395344 (+ 3603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:02:29.395353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:02:29.395369 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:02:29.395897 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:02:29.396295 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:02:29.396299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:02:29.396301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:02:29.396472 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:02:29.397079 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:02:29.397085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:02:29.397366 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 13:02:29.397595 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 13:02:29.398014 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":385,"threads_started":1}
I20260619 13:02:29.400429 13186 heartbeater.cc:347] Connected to a master server at 127.1.87.126:40409
I20260619 13:02:29.400775 13186 heartbeater.cc:464] Registering TS with master...
I20260619 13:02:29.401466 13186 heartbeater.cc:511] Master 127.1.87.126:40409 requested a full tablet report, sending...
I20260619 13:02:29.402956 12696 ts_manager.cc:194] Registered new tserver with Master: 89e5878dec0b4d1aacd41b187dce6547 (127.1.87.67:42523)
I20260619 13:02:29.403792 12696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:40601
I20260619 13:02:29.407718 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:02:29.425143 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.412032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.1.87.126:40409)
0619 13:02:29.412215 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:29.412224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:29.412976 (+ 752us) server_negotiation.cc:407] Connection header received
0619 13:02:29.413185 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:29.413190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:29.413270 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:29.413383 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:02:29.414680 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.415475 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.416821 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.417317 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.417446 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.419911 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:02:29.419917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:02:29.419921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:02:29.419964 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:02:29.422547 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:29.423198 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:29.423204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:29.423207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:29.423384 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:29.423768 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:29.423775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:29.423780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:29.424039 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:02:29.424297 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:29.424611 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 13:02:29.424822 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:02:29.428985 12696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:02:29.436128 12696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:02:29.463596 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.449482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.1.87.66:37295)
0619 13:02:29.451560 (+ 2078us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:29.451571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:29.454050 (+ 2479us) server_negotiation.cc:407] Connection header received
0619 13:02:29.454213 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:29.454223 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:29.454498 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:29.454716 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:02:29.455507 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.457089 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.460322 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.462134 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.462256 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.462446 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:29.463040 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 13:02:29.463228 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1735,"thread_start_us":1579,"threads_started":1}
I20260619 13:02:29.463727 13203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.449040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:37295 (local address 127.0.0.1:48708)
0619 13:02:29.453919 (+ 4879us) negotiation.cc:107] Waiting for socket to connect
0619 13:02:29.453936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:02:29.454075 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:02:29.454742 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:02:29.454750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:02:29.454784 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:02:29.455229 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.455257 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.457269 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.457274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.460160 (+ 2886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.460175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.462444 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.462452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.462711 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.462801 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 13:02:29.462933 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 13:02:29.463188 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4762,"mutex_wait_us":1212,"thread_start_us":3667,"threads_started":1}
I20260619 13:02:29.463749 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.449036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:35743 (local address 127.0.0.1:56406)
0619 13:02:29.451910 (+ 2874us) negotiation.cc:107] Waiting for socket to connect
0619 13:02:29.451957 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:02:29.452240 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:02:29.455026 (+ 2786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:02:29.455034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:02:29.455070 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:02:29.455661 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.455678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.457574 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.457582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.459919 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.459931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.462457 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.462463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.462711 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.462801 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 13:02:29.462933 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 13:02:29.463188 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":89,"spinlock_wait_cycles":1408,"thread_start_us":1179,"threads_started":1}
I20260619 13:02:29.464969 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.449078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.1.87.65:35743)
0619 13:02:29.449769 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:29.449778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:29.453985 (+ 4207us) server_negotiation.cc:407] Connection header received
0619 13:02:29.454125 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:29.454136 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:29.454836 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:29.455037 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:02:29.455868 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.457408 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.460094 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.462077 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.462239 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.462481 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:29.464433 (+ 1952us) server_negotiation.cc:299] Negotiation successful
0619 13:02:29.464612 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":405,"threads_started":1}
I20260619 13:02:29.465003 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.449922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:42523 (local address 127.0.0.1:47186)
0619 13:02:29.452851 (+ 2929us) negotiation.cc:107] Waiting for socket to connect
0619 13:02:29.452874 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:02:29.453011 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:02:29.455151 (+ 2140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:02:29.455155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:02:29.455179 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:02:29.455682 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.455690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.457708 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.457713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.460160 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.460174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.464180 (+ 4006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.464187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.464374 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.464410 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:02:29.464476 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 13:02:29.464611 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2809,"mutex_wait_us":98,"thread_start_us":2623,"threads_started":1}
I20260619 13:02:29.468030 12844 tablet_service.cc:1467] Processing CreateTablet for tablet 59b5278cd6224998a2c4dbbccd04f1b2 (DEFAULT_TABLE table=test-table [id=5346dc79a07a4eb98efe15ef8e7c28dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:02:29.468029 12987 tablet_service.cc:1467] Processing CreateTablet for tablet 59b5278cd6224998a2c4dbbccd04f1b2 (DEFAULT_TABLE table=test-table [id=5346dc79a07a4eb98efe15ef8e7c28dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:02:29.468951 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.450143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.1.87.67:42523)
0619 13:02:29.454614 (+ 4471us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:29.454623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:29.454654 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:02:29.454749 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:29.454756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:29.454952 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:29.455885 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:02:29.456105 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.457528 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.462544 (+ 5016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.463929 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.465285 (+ 1356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.465499 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:29.465566 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 13:02:29.465735 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4298,"thread_start_us":4146,"threads_started":1}
I20260619 13:02:29.469377 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b5278cd6224998a2c4dbbccd04f1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:02:29.469453 12844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b5278cd6224998a2c4dbbccd04f1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:02:29.472750 13121 tablet_service.cc:1467] Processing CreateTablet for tablet 59b5278cd6224998a2c4dbbccd04f1b2 (DEFAULT_TABLE table=test-table [id=5346dc79a07a4eb98efe15ef8e7c28dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:02:29.474268 13121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b5278cd6224998a2c4dbbccd04f1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:02:29.482267 13205 tablet_bootstrap.cc:492] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: Bootstrap starting.
I20260619 13:02:29.484656 13206 tablet_bootstrap.cc:492] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: Bootstrap starting.
I20260619 13:02:29.486514 13205 tablet_bootstrap.cc:654] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: Neither blocks nor log segments found. Creating new log.
I20260619 13:02:29.487874 13205 log.cc:824] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: Log is configured to *not* fsync() on all Append() calls
I20260619 13:02:29.488322 13206 tablet_bootstrap.cc:654] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: Neither blocks nor log segments found. Creating new log.
I20260619 13:02:29.489975 13206 log.cc:824] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: Log is configured to *not* fsync() on all Append() calls
I20260619 13:02:29.494247 13206 tablet_bootstrap.cc:492] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: No bootstrap required, opened a new log
I20260619 13:02:29.494777 13206 ts_tablet_manager.cc:1397] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 13:02:29.491833 13208 tablet_bootstrap.cc:492] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: Bootstrap starting.
I20260619 13:02:29.497304 13205 tablet_bootstrap.cc:492] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: No bootstrap required, opened a new log
I20260619 13:02:29.497586 13205 ts_tablet_manager.cc:1397] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260619 13:02:29.498751 13208 tablet_bootstrap.cc:654] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: Neither blocks nor log segments found. Creating new log.
I20260619 13:02:29.500051 13208 log.cc:824] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: Log is configured to *not* fsync() on all Append() calls
I20260619 13:02:29.503605 13208 tablet_bootstrap.cc:492] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: No bootstrap required, opened a new log
I20260619 13:02:29.503870 13208 ts_tablet_manager.cc:1397] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260619 13:02:29.505741 13206 raft_consensus.cc:348] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } }
I20260619 13:02:29.506193 13206 raft_consensus.cc:374] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:02:29.506345 13206 raft_consensus.cc:729] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cf8160326642be91bd34c57589f089, State: Initialized, Role: FOLLOWER
I20260619 13:02:29.506585 13205 raft_consensus.cc:348] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } }
I20260619 13:02:29.506958 13205 raft_consensus.cc:374] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:02:29.507081 13205 raft_consensus.cc:729] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22f91eb2f3741a2abedda9f62fe1c11, State: Initialized, Role: FOLLOWER
I20260619 13:02:29.507120 13206 consensus_queue.cc:260] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } }
I20260619 13:02:29.507674 13205 consensus_queue.cc:260] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } }
I20260619 13:02:29.508755 13052 heartbeater.cc:503] Master 127.1.87.126:40409 was elected leader, sending a full tablet report...
I20260619 13:02:29.509554 13205 ts_tablet_manager.cc:1428] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 13:02:29.512763 13206 ts_tablet_manager.cc:1428] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260619 13:02:29.515795 13208 raft_consensus.cc:348] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } }
I20260619 13:02:29.516183 13208 raft_consensus.cc:374] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:02:29.516297 13208 raft_consensus.cc:729] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e5878dec0b4d1aacd41b187dce6547, State: Initialized, Role: FOLLOWER
I20260619 13:02:29.516894 13208 consensus_queue.cc:260] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } }
I20260619 13:02:29.517970 13186 heartbeater.cc:503] Master 127.1.87.126:40409 was elected leader, sending a full tablet report...
I20260619 13:02:29.518606 13208 ts_tablet_manager.cc:1428] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260619 13:02:29.536505 13053 tablet.cc:2369] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:02:29.628470 12910 tablet.cc:2369] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:02:29.629385 13187 tablet.cc:2369] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:02:29.720937 13211 raft_consensus.cc:482] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:02:29.721297 13211 raft_consensus.cc:504] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } }
I20260619 13:02:29.723088 13211 leader_election.cc:288] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d22f91eb2f3741a2abedda9f62fe1c11 (127.1.87.66:37295), 89e5878dec0b4d1aacd41b187dce6547 (127.1.87.67:42523)
I20260619 13:02:29.730679 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.723710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:53641 (local address 127.1.87.66:37295)
0619 13:02:29.723985 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:29.723992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:29.725335 (+ 1343us) server_negotiation.cc:407] Connection header received
0619 13:02:29.725536 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:29.725541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:29.725620 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:29.725733 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:02:29.726334 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.727045 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.728900 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.729678 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.729849 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.729953 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:29.730255 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 13:02:29.730398 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:02:29.730772 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.723635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:37295 (local address 127.1.87.65:53641)
0619 13:02:29.725162 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0619 13:02:29.725219 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 13:02:29.725381 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:02:29.725750 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:02:29.725755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:02:29.725794 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:02:29.726169 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.726181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.727198 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.727204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.728739 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.728751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.729904 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.729910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.730066 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.730089 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:02:29.730233 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 13:02:29.730376 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1415,"thread_start_us":1171,"threads_started":1}
I20260619 13:02:29.730993 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.723872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:55029 (local address 127.1.87.67:42523)
0619 13:02:29.724101 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:29.724107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:29.725233 (+ 1126us) server_negotiation.cc:407] Connection header received
0619 13:02:29.725429 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:29.725441 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:29.725514 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:29.725633 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:02:29.726526 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.727225 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.729106 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.729969 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.730143 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.730272 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:29.730609 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 13:02:29.730740 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:02:29.731400 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.723476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:42523 (local address 127.1.87.65:55029)
0619 13:02:29.725060 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0619 13:02:29.725085 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:02:29.725259 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:02:29.725648 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:02:29.725654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:02:29.725679 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:02:29.726359 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.726371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.727397 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.727409 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.728927 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:02:29.728941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.730252 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.730261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:02:29.730429 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.730453 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:02:29.730584 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 13:02:29.730716 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"thread_start_us":565,"threads_started":1}
I20260619 13:02:29.732079 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b5278cd6224998a2c4dbbccd04f1b2" candidate_uuid: "c0cf8160326642be91bd34c57589f089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" is_pre_election: true
I20260619 13:02:29.732470 13141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b5278cd6224998a2c4dbbccd04f1b2" candidate_uuid: "c0cf8160326642be91bd34c57589f089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e5878dec0b4d1aacd41b187dce6547" is_pre_election: true
I20260619 13:02:29.732623 13007 raft_consensus.cc:2454] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0cf8160326642be91bd34c57589f089 in term 0.
I20260619 13:02:29.732949 13141 raft_consensus.cc:2454] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0cf8160326642be91bd34c57589f089 in term 0.
I20260619 13:02:29.733312 12797 leader_election.cc:302] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0cf8160326642be91bd34c57589f089, d22f91eb2f3741a2abedda9f62fe1c11; no voters:
I20260619 13:02:29.733728 13211 raft_consensus.cc:2781] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:02:29.733884 13211 raft_consensus.cc:482] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:02:29.733994 13211 raft_consensus.cc:3037] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:02:29.736563 13211 raft_consensus.cc:504] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } }
I20260619 13:02:29.737185 13211 leader_election.cc:288] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [CANDIDATE]: Term 1 election: Requested vote from peers d22f91eb2f3741a2abedda9f62fe1c11 (127.1.87.66:37295), 89e5878dec0b4d1aacd41b187dce6547 (127.1.87.67:42523)
I20260619 13:02:29.737655 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b5278cd6224998a2c4dbbccd04f1b2" candidate_uuid: "c0cf8160326642be91bd34c57589f089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22f91eb2f3741a2abedda9f62fe1c11"
I20260619 13:02:29.737752 13141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b5278cd6224998a2c4dbbccd04f1b2" candidate_uuid: "c0cf8160326642be91bd34c57589f089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e5878dec0b4d1aacd41b187dce6547"
I20260619 13:02:29.737875 13007 raft_consensus.cc:3037] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:02:29.737936 13141 raft_consensus.cc:3037] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:02:29.741266 13141 raft_consensus.cc:2454] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0cf8160326642be91bd34c57589f089 in term 1.
I20260619 13:02:29.741266 13007 raft_consensus.cc:2454] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0cf8160326642be91bd34c57589f089 in term 1.
I20260619 13:02:29.742128 12799 leader_election.cc:302] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e5878dec0b4d1aacd41b187dce6547, c0cf8160326642be91bd34c57589f089; no voters:
I20260619 13:02:29.742693 13211 raft_consensus.cc:2781] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:02:29.743225 13211 raft_consensus.cc:686] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 1 LEADER]: Becoming Leader. State: Replica: c0cf8160326642be91bd34c57589f089, State: Running, Role: LEADER
I20260619 13:02:29.743741 13211 consensus_queue.cc:237] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } }
I20260619 13:02:29.748461 12696 catalog_manager.cc:5495] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0cf8160326642be91bd34c57589f089 (127.1.87.65). New cstate: current_term: 1 leader_uuid: "c0cf8160326642be91bd34c57589f089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf8160326642be91bd34c57589f089" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 } health_report { overall_health: UNKNOWN } } }
I20260619 13:02:29.827976 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.822596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.1.87.65:35743)
0619 13:02:29.822783 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:29.822788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:29.822804 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:02:29.822893 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:29.822896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:29.822977 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:29.823101 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:02:29.823532 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.824300 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.825775 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.826183 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.826326 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.826837 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:02:29.827414 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:02:29.827570 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:29.827622 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 13:02:29.827714 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:02:29.835915 13141 raft_consensus.cc:1261] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [term 1 FOLLOWER]: Refusing update from remote peer c0cf8160326642be91bd34c57589f089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:02:29.835960 13007 raft_consensus.cc:1261] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [term 1 FOLLOWER]: Refusing update from remote peer c0cf8160326642be91bd34c57589f089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:02:29.836905 13216 consensus_queue.cc:1035] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d22f91eb2f3741a2abedda9f62fe1c11" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:02:29.837239 13211 consensus_queue.cc:1035] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e5878dec0b4d1aacd41b187dce6547" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:02:29.861690 13220 mvcc.cc:205] Tried to move back new op lower bound from 7298556517718495232 to 7298556517354491904. Current Snapshot: MvccSnapshot[applied={T|T < 7298556517718495232}]
I20260619 13:02:29.884675 12696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:02:29.884902 12696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:02:29.888561 12696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9adedf9dcf1a410db873dc182d8aed53: Sending DeleteTablet for 3 replicas of tablet 59b5278cd6224998a2c4dbbccd04f1b2
I20260619 13:02:29.889750 12844 tablet_service.cc:1514] Processing DeleteTablet for tablet 59b5278cd6224998a2c4dbbccd04f1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:02:29 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:56406
I20260619 13:02:29.889734 12987 tablet_service.cc:1514] Processing DeleteTablet for tablet 59b5278cd6224998a2c4dbbccd04f1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:02:29 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:48708
I20260619 13:02:29.890578 13121 tablet_service.cc:1514] Processing DeleteTablet for tablet 59b5278cd6224998a2c4dbbccd04f1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:02:29 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:47186
I20260619 13:02:29.891467 13231 tablet_replica.cc:330] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: stopping tablet replica
I20260619 13:02:29.891928 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 12776
I20260619 13:02:29.891916 13231 raft_consensus.cc:2229] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:02:29.892369 13231 raft_consensus.cc:2258] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:02:29.902909 13231 ts_tablet_manager.cc:1905] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:02:29.904215 13229 tablet_replica.cc:330] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: stopping tablet replica
I20260619 13:02:29.904774 13230 tablet_replica.cc:330] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: stopping tablet replica
I20260619 13:02:29.905740 13229 raft_consensus.cc:2229] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:02:29.905941 13230 raft_consensus.cc:2229] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:02:29.906673 13229 raft_consensus.cc:2258] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:02:29.908591 13230 raft_consensus.cc:2258] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:02:29.908850 13229 ts_tablet_manager.cc:1905] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:02:29.911194 13230 ts_tablet_manager.cc:1905] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:02:29.911187 13231 ts_tablet_manager.cc:1918] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:02:29.911371 13231 log.cc:1196] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/59b5278cd6224998a2c4dbbccd04f1b2
I20260619 13:02:29.911978 13231 ts_tablet_manager.cc:1939] T 59b5278cd6224998a2c4dbbccd04f1b2 P 89e5878dec0b4d1aacd41b187dce6547: Deleting consensus metadata
I20260619 13:02:29.913838 12682 catalog_manager.cc:4841] TS 89e5878dec0b4d1aacd41b187dce6547 (127.1.87.67:42523): tablet 59b5278cd6224998a2c4dbbccd04f1b2 (table test-table [id=5346dc79a07a4eb98efe15ef8e7c28dc]) successfully deleted
I20260619 13:02:29.918857 13229 ts_tablet_manager.cc:1918] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:02:29.919102 13229 log.cc:1196] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/59b5278cd6224998a2c4dbbccd04f1b2
I20260619 13:02:29.919749 13229 ts_tablet_manager.cc:1939] T 59b5278cd6224998a2c4dbbccd04f1b2 P c0cf8160326642be91bd34c57589f089: Deleting consensus metadata
Jun 19 13:02:29 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:02:29.919968 13230 ts_tablet_manager.cc:1918] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:02:29.920210 13230 log.cc:1196] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/59b5278cd6224998a2c4dbbccd04f1b2
I20260619 13:02:29.920851 13230 ts_tablet_manager.cc:1939] T 59b5278cd6224998a2c4dbbccd04f1b2 P d22f91eb2f3741a2abedda9f62fe1c11: Deleting consensus metadata
I20260619 13:02:29.921896 12681 catalog_manager.cc:4841] TS c0cf8160326642be91bd34c57589f089 (127.1.87.65:35743): tablet 59b5278cd6224998a2c4dbbccd04f1b2 (table test-table [id=5346dc79a07a4eb98efe15ef8e7c28dc]) successfully deleted
I20260619 13:02:29.922614 12680 catalog_manager.cc:4841] TS d22f91eb2f3741a2abedda9f62fe1c11 (127.1.87.66:37295): tablet 59b5278cd6224998a2c4dbbccd04f1b2 (table test-table [id=5346dc79a07a4eb98efe15ef8e7c28dc]) successfully deleted
I20260619 13:02:29.927909 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:29.892917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.1.87.65:35743)
0619 13:02:29.893133 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:29.893143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:29.905386 (+ 12243us) server_negotiation.cc:407] Connection header received
0619 13:02:29.906230 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:29.906236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:29.906323 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:29.906395 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:02:29.909095 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.910344 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.913305 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:29.913983 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:29.914125 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:29.921252 (+ 7127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:02:29.921269 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:02:29.921286 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:02:29.921333 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:02:29.924816 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:29.925651 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:29.925662 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:29.925668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:29.925827 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:29.926260 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:29.926263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:29.926265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:29.926737 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:02:29.926957 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:29.927400 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 13:02:29.927572 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":72}
I20260619 13:02:29.928633 12904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:02:30.030018 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 12776
I20260619 13:02:30.048274 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 12915
Jun 19 13:02:30 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:02:30.063933 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:30.049011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48716 (local address 127.1.87.66:37295)
0619 13:02:30.049247 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:30.049272 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:30.049294 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:02:30.049364 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:30.049368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:30.049436 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:30.049531 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:02:30.050875 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:30.051711 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:30.053130 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:30.053553 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:30.053669 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:30.057944 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:02:30.057959 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:02:30.057974 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:02:30.058007 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:02:30.061163 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:30.061900 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:30.061907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:30.061909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:30.062043 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:30.062521 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:30.062525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:30.062527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:30.062895 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:02:30.063125 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:30.063449 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 13:02:30.063653 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:02:30.064658 13047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:02:30.155761 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 12915
I20260619 13:02:30.172374 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 13058
Jun 19 13:02:30 dist-test-slave-tpsm krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:02:30.187199 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:02:30.173246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.1.87.67:42523)
0619 13:02:30.173421 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:02:30.173425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:02:30.173444 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:02:30.173585 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:02:30.173589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:02:30.173653 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:02:30.173749 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:02:30.174941 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:30.175714 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:30.176903 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:02:30.177278 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:02:30.177366 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:02:30.181425 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:02:30.181437 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:02:30.181453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:02:30.181484 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:02:30.184649 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:30.185304 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:30.185311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:30.185314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:30.185429 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:02:30.185803 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:02:30.185807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:02:30.185809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:02:30.186197 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:02:30.186412 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 13:02:30.186687 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 13:02:30.186837 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:02:30.187947 13181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:02:30.274874 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 13058
I20260619 13:02:30.290792 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 12664
I20260619 13:02:30.292261 12726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:02:30.404989 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 12664
I20260619 13:02:30.427588 1373 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 13:02:30.763681 1373 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 13:02:30.763939 1373 mini_ranger.cc:66] Stopping Ranger...
I20260619 13:02:31.109544 1373 mini_ranger.cc:68] Stopped Ranger
2026-06-19 13:02:31.109 UTC [12250] LOG: received smart shutdown request
2026-06-19 13:02:31.112 UTC [12250] LOG: background worker "logical replication launcher" (PID 12259) exited with exit code 1
2026-06-19 13:02:31.113 UTC [12254] LOG: shutting down
2026-06-19 13:02:31.271 UTC [12250] LOG: database system is shut down
2026-06-19T13:02:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49520 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:02:31 dist-test-slave-tpsm krb5kdc[13245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:02:31 dist-test-slave-tpsm krb5kdc[13245](info): set up 2 sockets
Jun 19 13:02:31 dist-test-slave-tpsm krb5kdc[13245](info): commencing operation
krb5kdc: starting...
W20260619 13:02:32.503794 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:02:32 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:02:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.87.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.87.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/rangeradmin_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/rangeradmin_127.1.87.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.87.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.87.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/rangerlookup_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/rangerlookup_127.1.87.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.87.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.87.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/HTTP_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/HTTP_127.1.87.84@KRBTEST.COM.keytab.
I20260619 13:02:32.809895 1373 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskas476m/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/postgres -l logfile start
2026-06-19 13:02:35.225 UTC [13275] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 13:02:35.225 UTC [13275] LOG: listening on IPv4 address "127.1.87.84", port 46629
2026-06-19 13:02:35.229 UTC [13275] LOG: listening on Unix socket "/tmp/.s.PGSQL.46629"
2026-06-19 13:02:35.251 UTC [13278] LOG: database system was shut down at 2026-06-19 13:02:33 UTC
2026-06-19 13:02:35.255 UTC [13275] LOG: database system is ready to accept connections
I20260619 13:02:36.227289 1373 mini_postgres.cc:96] Postgres bound to 46629
2026-06-19 13:02:36.248 UTC [13286] FATAL: database "slave" does not exist
127.1.87.84:46629 - accepting connections
I20260619 13:02:36.248826 1373 mini_ranger.cc:161] Starting Ranger...
I20260619 13:02:36.272940 1373 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 13:02:36.714972 1373 mini_ranger.cc:87] Created ranger Postgres database
I20260619 13:02:36.715083 1373 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-admin
2026-06-19 13:02:37,007 [I] DB FLAVOR :POSTGRES
2026-06-19 13:02:37,007 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:02:37,007 [I] Checking connection..
2026-06-19 13:02:37,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 13:02:37,242 [I] Checking connection passed.
2026-06-19 13:02:37,242 [I] --------- Verifying version history table ---------
2026-06-19 13:02:37,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:02:37,474 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:02:37,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:02:37,734 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:02:37,734 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 13:02:37,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 13:02:37.958 UTC [13350] WARNING: there is no transaction in progress
2026-06-19 13:02:37,965 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 13:02:37,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:02:38,234 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 13:02:38,234 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:02:38,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 13:02:38,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 13:02:38,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tpsm.c.gcp-upstream.internal','N') ;"
2026-06-19 13:02:38,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:02:39,203 [I] Table x_portal_user does not exist in database ranger
2026-06-19 13:02:39,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:02:39,464 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 13:02:39,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:02:39,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:02:39,946 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 13:02:39,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 13:02:40.882 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:40.897 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:40.913 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:40.927 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:40.942 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:40.959 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.016 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.022 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.037 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.048 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.059 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.073 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.083 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.090 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.096 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41.104 UTC [13458] WARNING: there is no transaction in progress
2026-06-19 13:02:41,697 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 13:02:41,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 13:02:41,951 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 13:02:41,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:02:42,197 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 13:02:42,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:02:42,447 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 13:02:42,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:02:42,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:02:42,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tpsm.c.gcp-upstream.internal';"
2026-06-19 13:02:43,170 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 13:02:43,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 13:02:43,423 [I] DB_PATCHES have already been applied
I20260619 13:02:43.429641 1373 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-admin:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskas476m/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 13:02:43.429731 1373 mini_ranger.cc:193] Using host: 127.1.87.84
I20260619 13:02:43.444492 1373 mini_ranger.cc:227] Ranger admin URL: http://127.1.87.84:51221
Jun 19, 2026 1:02:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:02:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:02:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:02:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:02:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:02:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:02:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 13:02:44.589648 1373 mini_ranger.cc:160] Time spent starting Ranger: real 8.341s user 0.004s sys 0.033s
Jun 19, 2026 1:03:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:03:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 1:03:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:03:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:03:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 13:03:05.650359 1373 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.87.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.87.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/rangerkms_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.87.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/rangerkms_127.1.87.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.87.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.87.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.87.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/HTTP_127.1.87.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.87.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/HTTP_127.1.87.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 13:03:05 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874185, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 13:03:05.811108 1373 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 13:03:05.830688 1373 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 13:03:06.535090 1373 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 13:03:06.535195 1373 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-kms
I20260619 13:03:06.535247 1373 mini_ranger_kms.cc:227] Using postgres at 127.1.87.84:46629
2026-06-19 13:03:06,899 [I] DB FLAVOR :POSTGRES
2026-06-19 13:03:06,899 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:03:06,899 [I] Checking connection
2026-06-19 13:03:06,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 13:03:07,138 [I] connection success
2026-06-19 13:03:07,139 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 13:03:07,139 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 13:03:07,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 13:03:07,372 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 13:03:07,372 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:03:07,373 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 13:03:07,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.87.84:46629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 13:03:07,624 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 13:03:08.079807 1373 mini_ranger_kms.cc:310] Created kms service
I20260619 13:03:08.270514 1373 mini_ranger_kms.cc:326] Created kudu user
I20260619 13:03:08.311347 1373 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 13:03:08 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874185, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:08.718531 1373 mini_ranger_kms.cc:384] Added ranger policy
I20260619 13:03:08.718851 1373 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-kms:/tmp/dist-test-taskas476m/build/asan/bin/postgresql.jar:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskas476m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 13:03:08.718902 1373 mini_ranger_kms.cc:240] Using host: 127.1.87.84
I20260619 13:03:08.736066 1373 mini_ranger_kms.cc:276] Ranger KMS PID: 13638
I20260619 13:03:08.736196 1373 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.87.84:41403
Jun 19, 2026 1:03:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:03:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:03:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:03:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:03:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:03:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 13:03:11.042512 1373 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.231s user 0.087s sys 0.290s
I20260619 13:03:11.042783 1373 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 13:03:11.042958 1373 mini_ranger_kms.cc:401] 127.1.87.84:41403/kms/v1/keys
Jun 19 13:03:11 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874191, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.87.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 13:03:11 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874191, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.87.84@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
Jun 19 13:03:11 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874191, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.87.84@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
Jun 19 13:03:13 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:03:14.063129 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:45513
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:45513
--ranger_config_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:41403/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:14.394646 13689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:14.395017 13689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:14.395192 13689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:14.402581 13689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:14.402765 13689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:14.402882 13689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:14.402981 13689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:03:14.403053 13689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:14.413663 13689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.87.84:41403/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:45513
--ranger_config_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:14.415068 13689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:14.416721 13689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:14.429170 13695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:14.429237 13697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:14.429276 13694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:14.430047 13689 server_base.cc:1034] running on GCE node
I20260619 13:03:14.430768 13689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:14.432205 13689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:14.433414 13689 hybrid_clock.cc:648] HybridClock initialized: now 1781874194433361 us; error 43 us; skew 500 ppm
I20260619 13:03:14.433791 13689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:14 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:14.436812 13689 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:03:14.438325 13689 webserver.cc:460] Webserver started at http://127.1.87.126:37151/ using document root <none> and password file <none>
I20260619 13:03:14.438968 13689 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:14.439102 13689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:14.439472 13689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:03:14 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:14.495393 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "4d2ef446470e4225874237487c775bd1"
format_stamp: "Formatted at 2026-06-19 13:03:14 on dist-test-slave-tpsm"
server_key: "49409d28f4438b57f23637efce5037f1"
server_key_iv: "f2ba9ef07e079bdc54105fb2da27bc7c"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:14.496273 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "4d2ef446470e4225874237487c775bd1"
format_stamp: "Formatted at 2026-06-19 13:03:14 on dist-test-slave-tpsm"
server_key: "49409d28f4438b57f23637efce5037f1"
server_key_iv: "f2ba9ef07e079bdc54105fb2da27bc7c"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:14.501729 13689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:03:14.517575 13704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:14.519114 13689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 13:03:14.519301 13689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "4d2ef446470e4225874237487c775bd1"
format_stamp: "Formatted at 2026-06-19 13:03:14 on dist-test-slave-tpsm"
server_key: "49409d28f4438b57f23637efce5037f1"
server_key_iv: "f2ba9ef07e079bdc54105fb2da27bc7c"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:14.519577 13689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:14.541213 13689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:14.545228 13689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:14.545560 13689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:14.565542 13689 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:45513
I20260619 13:03:14.565562 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:45513 every 8 connection(s)
I20260619 13:03:14.566895 13689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:03:14.569708 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 13689
I20260619 13:03:14.569916 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:03:14.570745 1373 external_mini_cluster.cc:1428] Setting key 636ab702de69a17dd81c1dc5e47a1ddb
I20260619 13:03:14.571511 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:14.586117 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Bootstrap starting.
Jun 19 13:03:14 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:14.589660 13756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:14.590785 13756 log.cc:824] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:14.594264 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: No bootstrap required, opened a new log
I20260619 13:03:14.594705 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:14.572914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.1.87.126:45513)
0619 13:03:14.573660 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:14.573671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:14.573710 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:03:14.576030 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:14.576052 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:14.576445 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:14.576771 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:14.578463 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:14.579925 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:14.581740 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:14.582487 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:14.582626 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:14.587518 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:14.587527 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:14.587546 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:14.587600 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:14.590669 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:14.591604 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:14.591612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:14.591621 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:14.591833 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:14.592187 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:14.592193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:14.592196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:14.592697 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:14.592911 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:14.593378 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 13:03:14.593697 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":364,"threads_started":1}
I20260619 13:03:14.602236 13756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } }
I20260619 13:03:14.602622 13756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:14.602747 13756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2ef446470e4225874237487c775bd1, State: Initialized, Role: FOLLOWER
I20260619 13:03:14.603292 13756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } }
I20260619 13:03:14.603633 13756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:03:14.603791 13756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:03:14.603963 13756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:14.606429 13756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } }
I20260619 13:03:14.606930 13756 leader_election.cc:302] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d2ef446470e4225874237487c775bd1; no voters:
I20260619 13:03:14.607540 13756 leader_election.cc:288] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:03:14.607671 13761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:03:14.608397 13761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 1 LEADER]: Becoming Leader. State: Replica: 4d2ef446470e4225874237487c775bd1, State: Running, Role: LEADER
I20260619 13:03:14.608958 13761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } }
I20260619 13:03:14.609720 13756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:03:14.611840 13763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d2ef446470e4225874237487c775bd1. Latest consensus state: current_term: 1 leader_uuid: "4d2ef446470e4225874237487c775bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } } }
I20260619 13:03:14.612313 13763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:14.613597 13762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d2ef446470e4225874237487c775bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } } }
I20260619 13:03:14.613867 13762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:14.614144 13769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:03:14.619611 13769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:03:14.620241 13756 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 13:03:14.638046 13769 catalog_manager.cc:1347] Generated new cluster ID: bbfcd9f9454947a48d836707aa8c9f47
I20260619 13:03:14.638289 13769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:03:14.659972 13769 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:03:14.662194 13769 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:03:14.669797 13769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Generated new TSK 0
I20260619 13:03:14.670872 13769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:03:15.425448 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:41403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45513
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:03:15 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 13:03:15.810539 13801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:15.810853 13801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:15.811023 13801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:15.819916 13801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:15.820063 13801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:15.820246 13801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:03:15.833006 13801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.87.84:41403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:15.834614 13801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:15.836282 13801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:15.852200 13811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:15.853910 13812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:15.861964 13814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:15.862705 13801 server_base.cc:1034] running on GCE node
I20260619 13:03:15.863595 13801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:15.864907 13801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:15.866207 13801 hybrid_clock.cc:648] HybridClock initialized: now 1781874195866146 us; error 63 us; skew 500 ppm
I20260619 13:03:15.866603 13801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:15 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:15.870886 13801 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:03:15.883656 13801 webserver.cc:460] Webserver started at http://127.1.87.65:45749/ using document root <none> and password file <none>
I20260619 13:03:15.884678 13801 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:15.884865 13801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:15.885380 13801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:03:15 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:15.905323 13801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "f2872e3dfd774891873632d7bffd8d6a"
format_stamp: "Formatted at 2026-06-19 13:03:15 on dist-test-slave-tpsm"
server_key: "b8d866c1220843da90db0cbd286c79bd"
server_key_iv: "8a4da5860f7923aaef145e4d1987b794"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:15.909904 13801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "f2872e3dfd774891873632d7bffd8d6a"
format_stamp: "Formatted at 2026-06-19 13:03:15 on dist-test-slave-tpsm"
server_key: "b8d866c1220843da90db0cbd286c79bd"
server_key_iv: "8a4da5860f7923aaef145e4d1987b794"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:15.920805 13801 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s
I20260619 13:03:15.953616 13821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:15.956629 13801 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.009s sys 0.001s
I20260619 13:03:15.956926 13801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "f2872e3dfd774891873632d7bffd8d6a"
format_stamp: "Formatted at 2026-06-19 13:03:15 on dist-test-slave-tpsm"
server_key: "b8d866c1220843da90db0cbd286c79bd"
server_key_iv: "8a4da5860f7923aaef145e4d1987b794"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:15.957324 13801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:15.983086 13801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:15.986943 13801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:15.987253 13801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:15.988391 13801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:15.989804 13801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:15.989874 13801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:15.989998 13801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:15.990060 13801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:16.046808 13801 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:45991
I20260619 13:03:16.046847 13933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:45991 every 8 connection(s)
I20260619 13:03:16.048573 13801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:03:16.051369 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 13801
I20260619 13:03:16.051576 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:03:16.052402 1373 external_mini_cluster.cc:1428] Setting key 92f24ceb082269f0baf1269702465397
Jun 19 13:03:16 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:16.091645 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:16.057727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:39533 (local address 127.1.87.126:45513)
0619 13:03:16.066365 (+ 8638us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:16.066372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:16.073015 (+ 6643us) server_negotiation.cc:407] Connection header received
0619 13:03:16.075053 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:16.075058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:16.075152 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:16.075283 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:16.076830 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:16.077579 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:16.079394 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:16.079812 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:16.079948 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:16.083706 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:16.083715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:16.083723 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:16.083773 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:16.087623 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:16.088592 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:16.088600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:16.088607 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:16.088754 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:16.089316 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:16.089324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:16.089330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:16.089706 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:16.089845 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:16.090787 (+ 942us) server_negotiation.cc:299] Negotiation successful
0619 13:03:16.091093 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8505,"thread_start_us":8368,"threads_started":1}
I20260619 13:03:16.094329 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:16.058370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45513 (local address 127.1.87.65:39533)
0619 13:03:16.072840 (+ 14470us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:16.072888 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:16.074715 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:16.075541 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:16.075550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:16.076081 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:16.076656 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:16.076672 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:16.077745 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:16.077749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:16.079219 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:16.079232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:16.080041 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:16.080048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:16.080268 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:16.080945 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:16.080965 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:16.083461 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:16.087789 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:16.087800 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:16.087821 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:16.088444 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:16.088896 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:16.088903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:16.088909 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:16.089161 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:16.089942 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:16.089953 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:16.090377 (+ 424us) client_negotiation.cc:769] Sending connection context
0619 13:03:16.092311 (+ 1934us) client_negotiation.cc:240] Negotiation successful
0619 13:03:16.092869 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14140,"thread_start_us":13935,"threads_started":1}
I20260619 13:03:16.096552 13934 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45513
I20260619 13:03:16.096925 13934 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:16.097872 13934 heartbeater.cc:511] Master 127.1.87.126:45513 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
I20260619 13:03:16.100586 13721 ts_manager.cc:194] Registered new tserver with Master: f2872e3dfd774891873632d7bffd8d6a (127.1.87.65:45991)
I20260619 13:03:16.102788 13721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:39533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:03:16.130291 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:41403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45513
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:03:16 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
Jun 19 13:03:16 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
W20260619 13:03:16.595119 13940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:16.595672 13940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:16.595875 13940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:16.607501 13940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:16.607731 13940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:16.607964 13940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:03:16.624240 13940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.87.84:41403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:16.625877 13940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:16.627444 13940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:16.646562 13948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:16.646657 13947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:16.648079 13950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:16.651799 13940 server_base.cc:1034] running on GCE node
I20260619 13:03:16.652634 13940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:16.654367 13940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:16.655656 13940 hybrid_clock.cc:648] HybridClock initialized: now 1781874196655590 us; error 65 us; skew 500 ppm
I20260619 13:03:16.656044 13940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:16 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:16.660425 13940 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:03:16.715262 13940 webserver.cc:460] Webserver started at http://127.1.87.66:41133/ using document root <none> and password file <none>
I20260619 13:03:16.716612 13940 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:16.716833 13940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:16.717245 13940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:03:16 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:16.736056 13940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "667d939964c449a98ed79c582d34e5f7"
format_stamp: "Formatted at 2026-06-19 13:03:16 on dist-test-slave-tpsm"
server_key: "8b5fd833f85fb72fa9f94f80a295721e"
server_key_iv: "f1815ce02c5c714b04cd687bf5b70c40"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:16.737241 13940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "667d939964c449a98ed79c582d34e5f7"
format_stamp: "Formatted at 2026-06-19 13:03:16 on dist-test-slave-tpsm"
server_key: "8b5fd833f85fb72fa9f94f80a295721e"
server_key_iv: "f1815ce02c5c714b04cd687bf5b70c40"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:16.745828 13940 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.003s
I20260619 13:03:16.781008 13957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:16.784817 13940 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.003s
I20260619 13:03:16.785138 13940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "667d939964c449a98ed79c582d34e5f7"
format_stamp: "Formatted at 2026-06-19 13:03:16 on dist-test-slave-tpsm"
server_key: "8b5fd833f85fb72fa9f94f80a295721e"
server_key_iv: "f1815ce02c5c714b04cd687bf5b70c40"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:16.785499 13940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:16.837373 13940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:16.842927 13940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:16.843394 13940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:16.859787 13940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:16.868340 13940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:16.868543 13940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:16.868722 13940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:16.868829 13940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:16.992365 13940 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:43527
I20260619 13:03:16.992420 14076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:43527 every 8 connection(s)
I20260619 13:03:16.993840 13940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:03:17.003376 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 13940
I20260619 13:03:17.003582 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:03:17.004230 1373 external_mini_cluster.cc:1428] Setting key a175f219d2759d0583d365aa88bf5834
Jun 19 13:03:17 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:17.038024 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:16.997044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:58433 (local address 127.1.87.126:45513)
0619 13:03:17.012981 (+ 15937us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:17.012987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:17.013011 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:03:17.016838 (+ 3827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:17.016846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:17.016956 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:17.017053 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:17.019488 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.020676 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.023629 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.024472 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.024592 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:17.029486 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:17.029497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:17.029505 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:17.029554 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:17.033599 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.034658 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.034667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.034674 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.034878 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.035581 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.035587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.035593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.035995 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:17.036146 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:17.037323 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0619 13:03:17.037588 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15802,"thread_start_us":15969,"threads_started":1}
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:03:17.040721 14080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:16.997828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45513 (local address 127.1.87.66:58433)
0619 13:03:17.011274 (+ 13446us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:17.012541 (+ 1267us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:17.016421 (+ 3880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:17.017574 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:17.017588 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:17.018319 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:17.019213 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:17.019243 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.020874 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.020882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:17.023453 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:17.023471 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.024955 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.024964 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:17.025307 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:17.026111 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:17.026154 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:17.029197 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:17.033784 (+ 4587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:17.033797 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:17.033820 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:17.034505 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:17.035050 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:17.035055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:17.035058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:17.035429 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:17.036265 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:17.036278 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:17.036802 (+ 524us) client_negotiation.cc:769] Sending connection context
0619 13:03:17.038653 (+ 1851us) client_negotiation.cc:240] Negotiation successful
0619 13:03:17.039244 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12974,"thread_start_us":13460,"threads_started":1}
I20260619 13:03:17.042302 14077 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
I20260619 13:03:17.042618 14077 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:17.043370 14077 heartbeater.cc:511] Master 127.1.87.126:45513 requested a full tablet report, sending...
I20260619 13:03:17.048702 13721 ts_manager.cc:194] Registered new tserver with Master: 667d939964c449a98ed79c582d34e5f7 (127.1.87.66:43527)
I20260619 13:03:17.050037 13721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:58433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:03:17.076640 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:41403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45513
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:03:17.106731 13934 heartbeater.cc:503] Master 127.1.87.126:45513 was elected leader, sending a full tablet report...
W20260619 13:03:17.478482 14083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:17.478760 14083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:17.478909 14083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:17.486678 14083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:17.486797 14083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:17.486999 14083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:03:17.497325 14083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.87.84:41403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:17.498767 14083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:17.500092 14083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:17.513373 14089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:17.513449 14088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:17.513446 14091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:17.513990 14083 server_base.cc:1034] running on GCE node
I20260619 13:03:17.514616 14083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:17.515703 14083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:17.517074 14083 hybrid_clock.cc:648] HybridClock initialized: now 1781874197517007 us; error 57 us; skew 500 ppm
I20260619 13:03:17.517469 14083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:17 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:17.520813 14083 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:03:17.522202 14083 webserver.cc:460] Webserver started at http://127.1.87.67:41025/ using document root <none> and password file <none>
I20260619 13:03:17.523054 14083 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:17.523168 14083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:17.523500 14083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:03:17 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:17.541880 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "ced20bed01cd4bf583da68c21547aca1"
format_stamp: "Formatted at 2026-06-19 13:03:17 on dist-test-slave-tpsm"
server_key: "1730b3f50ca3028d49a1445975af8cce"
server_key_iv: "f661bc9dda8a2fca01d47e9e0fe48945"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:17.542886 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "ced20bed01cd4bf583da68c21547aca1"
format_stamp: "Formatted at 2026-06-19 13:03:17 on dist-test-slave-tpsm"
server_key: "1730b3f50ca3028d49a1445975af8cce"
server_key_iv: "f661bc9dda8a2fca01d47e9e0fe48945"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:17.548609 14083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:03:17.566476 14098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:17.568074 14083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:03:17.568264 14083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "ced20bed01cd4bf583da68c21547aca1"
format_stamp: "Formatted at 2026-06-19 13:03:17 on dist-test-slave-tpsm"
server_key: "1730b3f50ca3028d49a1445975af8cce"
server_key_iv: "f661bc9dda8a2fca01d47e9e0fe48945"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:17.568499 14083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:17.583142 14083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:17.586959 14083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:17.587266 14083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:17.588236 14083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:17.589509 14083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:17.589582 14083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:17.589666 14083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:17.589708 14083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:17.623179 14083 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:45815
I20260619 13:03:17.623204 14210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:45815 every 8 connection(s)
I20260619 13:03:17.624555 14083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:03:17.625156 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 14083
I20260619 13:03:17.625373 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:03:17.626014 1373 external_mini_cluster.cc:1428] Setting key 3d1a99df268928a7638b6e735f85a6e4
Jun 19 13:03:17 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:17.644963 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:17.627137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:34405 (local address 127.1.87.126:45513)
0619 13:03:17.627658 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:17.627687 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:17.628716 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 13:03:17.630585 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:17.630591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:17.630685 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:17.630803 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:17.632240 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.632972 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.634485 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.634934 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.635061 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:17.638566 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:17.638571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:17.638576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:17.638617 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:17.642010 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.642865 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.642869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.642871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.642987 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.643388 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.643393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.643397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.643670 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:17.643780 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:17.644525 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 13:03:17.644708 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":260,"threads_started":1}
I20260619 13:03:17.645834 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:17.627795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45513 (local address 127.1.87.67:34405)
0619 13:03:17.628581 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:17.628620 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:17.630333 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:17.631025 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:17.631033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:17.631528 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:17.632083 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:17.632099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.633139 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.633150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:17.634344 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:17.634356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.635176 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.635184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:17.635525 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:17.636241 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:17.636262 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:17.638368 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:17.642290 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:17.642297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:17.642311 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:17.642759 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:17.643082 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:17.643084 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:17.643087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:17.643249 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:17.643811 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:17.643817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:17.644130 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 13:03:17.644371 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 13:03:17.644744 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":368,"threads_started":1}
I20260619 13:03:17.647022 14211 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45513
I20260619 13:03:17.647369 14211 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:17.648049 14211 heartbeater.cc:511] Master 127.1.87.126:45513 requested a full tablet report, sending...
I20260619 13:03:17.649490 13721 ts_manager.cc:194] Registered new tserver with Master: ced20bed01cd4bf583da68c21547aca1 (127.1.87.67:45815)
I20260619 13:03:17.650285 13721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:34405
I20260619 13:03:17.654460 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:03:17.656255 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 13801
Jun 19 13:03:17 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:03:17.672580 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:17.657143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.1.87.65:45991)
0619 13:03:17.657825 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:17.657832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:17.657854 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:03:17.657937 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:17.657943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:17.658119 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:17.658247 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:17.659860 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.660860 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.662297 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.662832 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.662925 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:17.667178 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:17.667188 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:17.667202 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:17.667238 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:17.669883 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.670607 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.670614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.670616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.670780 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.671106 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.671109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.671111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.671501 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:17.671776 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:17.672057 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 13:03:17.672301 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":412,"threads_started":1}
I20260619 13:03:17.673900 13929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:17.756982 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 13801
I20260619 13:03:17.774370 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 13940
Jun 19 13:03:17 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:03:17.789601 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:17.775233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.1.87.66:43527)
0619 13:03:17.775842 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:17.775849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:17.775873 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:03:17.775946 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:17.775953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:17.776150 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:17.776288 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:17.777451 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.778439 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.779712 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.780245 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.780340 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:17.784596 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:17.784608 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:17.784622 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:17.784654 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:17.787192 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.787849 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.787855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.787858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.787993 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.788300 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.788304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.788306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.788636 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:17.788845 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:17.789086 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 13:03:17.789294 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":350,"threads_started":1}
I20260619 13:03:17.791070 14072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:17.873449 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 13940
I20260619 13:03:17.888890 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 14083
Jun 19 13:03:17 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:03:17.905695 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:17.889871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.1.87.67:45815)
0619 13:03:17.890442 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:17.890449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:17.890481 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:03:17.890553 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:17.890560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:17.890781 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:17.890941 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:17.892429 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.893521 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.894947 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:17.895635 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:17.895761 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:17.900249 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:17.900260 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:17.900274 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:17.900310 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:17.902914 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.903650 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.903656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.903659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.903788 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:17.904156 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:17.904166 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:17.904168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:17.904495 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:17.904816 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:17.905087 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 13:03:17.905358 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":291,"threads_started":1}
I20260619 13:03:17.906947 14206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:17.991209 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 14083
I20260619 13:03:18.006583 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 13689
I20260619 13:03:18.007906 13751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:18.115808 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 13689
I20260619 13:03:18.136904 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:45513
--webserver_interface=127.1.87.126
--webserver_port=37151
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:45513
--ranger_config_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:41403/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.87.84:41403/kms with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:18.461614 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:18.461907 14223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:18.461997 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:18.471779 14223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:18.471946 14223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:18.472029 14223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:18.472121 14223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:03:18.472188 14223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:18.483740 14223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.87.84:41403/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:45513
--ranger_config_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=37151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:18.485025 14223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:18.486501 14223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:18.498428 14231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:18.498467 14229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:18.498502 14228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:18.499686 14223 server_base.cc:1034] running on GCE node
I20260619 13:03:18.500451 14223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:18.501875 14223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:18.503111 14223 hybrid_clock.cc:648] HybridClock initialized: now 1781874198503082 us; error 52 us; skew 500 ppm
I20260619 13:03:18.503545 14223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:18 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:18.510474 14223 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:03:18.512172 14223 webserver.cc:460] Webserver started at http://127.1.87.126:37151/ using document root <none> and password file <none>
I20260619 13:03:18.512876 14223 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:18.513038 14223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:03:18 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:18.561647 14223 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:03:18.564695 14239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:18.566177 14223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:18.566397 14223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "4d2ef446470e4225874237487c775bd1"
format_stamp: "Formatted at 2026-06-19 13:03:14 on dist-test-slave-tpsm"
server_key: "49409d28f4438b57f23637efce5037f1"
server_key_iv: "f2ba9ef07e079bdc54105fb2da27bc7c"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:18.567385 14223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:18.589721 14223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:18.594045 14223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:18.594341 14223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:18.613921 14223 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:45513
I20260619 13:03:18.613955 14290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:45513 every 8 connection(s)
I20260619 13:03:18.615612 14223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:03:18.621738 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 14223
I20260619 13:03:18.622795 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:41403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:45991
--local_ip_for_outbound_sockets=127.1.87.65
--tserver_master_addrs=127.1.87.126:45513
--webserver_port=45749
--webserver_interface=127.1.87.65
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.87.84:41403/kms with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:03:18.622963 14291 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:03:18.628252 14291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Bootstrap starting.
I20260619 13:03:18.647307 14291 log.cc:824] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:18.656672 14291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:03:18.657209 14291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Bootstrap complete.
I20260619 13:03:18.666543 14291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } }
I20260619 13:03:18.667492 14291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2ef446470e4225874237487c775bd1, State: Initialized, Role: FOLLOWER
I20260619 13:03:18.668114 14291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } }
I20260619 13:03:18.668344 14291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:03:18.668467 14291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:03:18.668653 14291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:03:18.672785 14291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } }
I20260619 13:03:18.673280 14291 leader_election.cc:302] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d2ef446470e4225874237487c775bd1; no voters:
I20260619 13:03:18.673878 14291 leader_election.cc:288] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:03:18.674124 14295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:03:18.675132 14295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [term 2 LEADER]: Becoming Leader. State: Replica: 4d2ef446470e4225874237487c775bd1, State: Running, Role: LEADER
I20260619 13:03:18.675949 14291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:03:18.675879 14295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } }
I20260619 13:03:18.686038 14296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4d2ef446470e4225874237487c775bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } } }
I20260619 13:03:18.686527 14296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:18.688932 14297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d2ef446470e4225874237487c775bd1. Latest consensus state: current_term: 2 leader_uuid: "4d2ef446470e4225874237487c775bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2ef446470e4225874237487c775bd1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45513 } } }
I20260619 13:03:18.689244 14297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:18.692296 14307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:03:18.697824 14307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:03:18.699415 14307 catalog_manager.cc:1259] Loaded cluster ID: bbfcd9f9454947a48d836707aa8c9f47
I20260619 13:03:18.699574 14307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:03:18.703586 14307 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:03:18.705161 14307 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Loaded TSK: 0
I20260619 13:03:18.706776 14307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:03:18.709399 14291 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 13:03:19.059625 14293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:19.060158 14293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:19.060359 14293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:19.070796 14293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:19.071033 14293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:19.071302 14293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:03:19.087970 14293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.87.84:41403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:45991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=45749
--tserver_master_addrs=127.1.87.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:19.089383 14293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:19.090809 14293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:19.138144 14330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:19.142472 14329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:19.147423 14332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:19.147895 14293 server_base.cc:1034] running on GCE node
I20260619 13:03:19.148856 14293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:19.150204 14293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:19.153607 14293 hybrid_clock.cc:648] HybridClock initialized: now 1781874199153515 us; error 57 us; skew 500 ppm
I20260619 13:03:19.154021 14293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:19 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:19.157670 14293 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:03:19.163935 14293 webserver.cc:460] Webserver started at http://127.1.87.65:45749/ using document root <none> and password file <none>
I20260619 13:03:19.164819 14293 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:19.164994 14293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:03:19 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:19.201676 14293 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.007s
I20260619 13:03:19.212306 14340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:19.214114 14293 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.007s
I20260619 13:03:19.214349 14293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "f2872e3dfd774891873632d7bffd8d6a"
format_stamp: "Formatted at 2026-06-19 13:03:15 on dist-test-slave-tpsm"
server_key: "b8d866c1220843da90db0cbd286c79bd"
server_key_iv: "8a4da5860f7923aaef145e4d1987b794"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:19.215858 14293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:19.266431 14293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:19.271235 14293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:19.271574 14293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:19.275815 14293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:19.290375 14293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:19.290575 14293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:19.290791 14293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:19.290935 14293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:19.363372 14293 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:45991
I20260619 13:03:19.363763 14453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:45991 every 8 connection(s)
I20260619 13:03:19.365294 14293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:03:19.365866 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 14293
I20260619 13:03:19.366788 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:41403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:43527
--local_ip_for_outbound_sockets=127.1.87.66
--tserver_master_addrs=127.1.87.126:45513
--webserver_port=41133
--webserver_interface=127.1.87.66
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.87.84:41403/kms with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:03:19 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:19.415766 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:19.373652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:57269 (local address 127.1.87.126:45513)
0619 13:03:19.378942 (+ 5290us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:19.378960 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:19.379007 (+ 47us) server_negotiation.cc:407] Connection header received
0619 13:03:19.379445 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:19.379462 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:19.379893 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:19.380283 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:19.382135 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:19.384250 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:19.387214 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:19.388556 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:19.388778 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:19.402458 (+ 13680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:19.402472 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:19.402495 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:19.402551 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:19.410006 (+ 7455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:19.411218 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:19.411229 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:19.411237 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:19.411490 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:19.412090 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:19.412095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:19.412098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:19.412786 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:19.412981 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:19.414066 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0619 13:03:19.414467 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4915,"thread_start_us":4662,"threads_started":1}
I20260619 13:03:19.415776 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:19.372010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45513 (local address 127.1.87.65:57269)
0619 13:03:19.375588 (+ 3578us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:19.375643 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:19.378022 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:19.380476 (+ 2454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:19.380489 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:19.381137 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:19.381882 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:19.381909 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:19.384494 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:19.384500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:19.386753 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:19.386775 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:19.389226 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:19.389235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:19.389696 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:19.398947 (+ 9251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:19.399002 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:19.402033 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:19.410276 (+ 8243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:19.410287 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:19.410306 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:19.411014 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:19.411667 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:19.411672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:19.411676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:19.411937 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:19.413084 (+ 1147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:19.413092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:19.413479 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 13:03:19.413830 (+ 351us) client_negotiation.cc:240] Negotiation successful
0619 13:03:19.414406 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3057,"thread_start_us":2611,"threads_started":1}
I20260619 13:03:19.420506 14454 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45513
I20260619 13:03:19.420854 14454 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:19.421743 14454 heartbeater.cc:511] Master 127.1.87.126:45513 requested a full tablet report, sending...
W20260619 13:03:19.425048 14454 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:45513 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 13:03:19.425272 14454 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:19.425544 14454 heartbeater.cc:511] Master 127.1.87.126:45513 requested a full tablet report, sending...
I20260619 13:03:19.426941 14454 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:19.427337 14454 heartbeater.cc:511] Master 127.1.87.126:45513 requested a full tablet report, sending...
W20260619 13:03:19.428686 14454 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:03:19.757961 14458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:19.758298 14458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:19.758473 14458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:19.766251 14458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:19.766405 14458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:19.766600 14458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:03:19.777372 14458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.87.84:41403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:43527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=41133
--tserver_master_addrs=127.1.87.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:19.778746 14458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:19.780153 14458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:19.797768 14470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:19.797801 14471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:19.800141 14458 server_base.cc:1034] running on GCE node
W20260619 13:03:19.800522 14473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:19.801270 14458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:19.802312 14458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:19.803560 14458 hybrid_clock.cc:648] HybridClock initialized: now 1781874199803501 us; error 54 us; skew 500 ppm
I20260619 13:03:19.803923 14458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:19 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:19.807475 14458 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:03:19.811344 14458 webserver.cc:460] Webserver started at http://127.1.87.66:41133/ using document root <none> and password file <none>
I20260619 13:03:19.812112 14458 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:19.812254 14458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:03:19 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:19.836862 14458 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:03:19.841490 14483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:19.843425 14458 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:03:19.843632 14458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "667d939964c449a98ed79c582d34e5f7"
format_stamp: "Formatted at 2026-06-19 13:03:16 on dist-test-slave-tpsm"
server_key: "8b5fd833f85fb72fa9f94f80a295721e"
server_key_iv: "f1815ce02c5c714b04cd687bf5b70c40"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:19.844996 14458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:19.874338 14458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:19.879647 14458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:19.879961 14458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:19.881196 14458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:19.882870 14458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:19.882984 14458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:19.883095 14458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:19.883170 14458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:19.945133 14458 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:43527
I20260619 13:03:19.945159 14596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:43527 every 8 connection(s)
I20260619 13:03:19.947139 14458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:03:19.954834 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 14458
I20260619 13:03:19.955729 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.87.84:41403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:45815
--local_ip_for_outbound_sockets=127.1.87.67
--tserver_master_addrs=127.1.87.126:45513
--webserver_port=41025
--webserver_interface=127.1.87.67
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.87.84:41403/kms with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:03:19 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:19.991178 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:19.963003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:36535 (local address 127.1.87.126:45513)
0619 13:03:19.967684 (+ 4681us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:19.967690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:19.967714 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:03:19.970569 (+ 2855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:19.970575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:19.970704 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:19.970791 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:19.973062 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:19.974256 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:19.976729 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:19.977436 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:19.977550 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:19.983772 (+ 6222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:19.983780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:19.983784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:19.983830 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:19.987406 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:19.988362 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:19.988368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:19.988371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:19.988529 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:19.989080 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:19.989085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:19.989088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:19.989470 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:19.989632 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:19.990563 (+ 931us) server_negotiation.cc:299] Negotiation successful
0619 13:03:19.990811 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4526,"thread_start_us":4560,"threads_started":1}
I20260619 13:03:19.993381 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:19.962363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45513 (local address 127.1.87.66:36535)
0619 13:03:19.967144 (+ 4781us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:19.967203 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:19.970222 (+ 3019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:19.971391 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:19.971405 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:19.972026 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:19.972855 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:19.972880 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:19.974460 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:19.974466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:19.976550 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:19.976564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:19.977940 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:19.977948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:19.978242 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:19.979050 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:19.979081 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:19.983043 (+ 3962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:19.987600 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:19.987610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:19.987628 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:19.988216 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:19.988696 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:19.988702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:19.988706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:19.988940 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:19.989734 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:19.989742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:19.990132 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 13:03:19.991423 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0619 13:03:19.992001 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4434,"thread_start_us":4189,"threads_started":1}
I20260619 13:03:19.995204 14598 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45513
I20260619 13:03:19.995570 14598 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:19.996410 14598 heartbeater.cc:511] Master 127.1.87.126:45513 requested a full tablet report, sending...
I20260619 13:03:19.998541 14256 ts_manager.cc:194] Registered new tserver with Master: 667d939964c449a98ed79c582d34e5f7 (127.1.87.66:43527)
I20260619 13:03:20.000833 14256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:36535
Jun 19 13:03:20 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 13:03:20.397500 14601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:20.397939 14601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:20.398144 14601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:20.406500 14601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:20.406647 14601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:20.406862 14601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:03:20.421434 14601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.87.84:41403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:45815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=41025
--tserver_master_addrs=127.1.87.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:20.423131 14601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:20.424826 14601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:03:20.428831 14454 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:20.429230 14454 heartbeater.cc:511] Master 127.1.87.126:45513 requested a full tablet report, sending...
I20260619 13:03:20.430356 14256 ts_manager.cc:194] Registered new tserver with Master: f2872e3dfd774891873632d7bffd8d6a (127.1.87.65:45991)
I20260619 13:03:20.431577 14256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:57269
Jun 19 13:03:20 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
W20260619 13:03:20.452049 14608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:20.465459 14601 server_base.cc:1034] running on GCE node
W20260619 13:03:20.471511 14611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:20.472507 14609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:20.473459 14601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:20.479506 14601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:20.482889 14601 hybrid_clock.cc:648] HybridClock initialized: now 1781874200482532 us; error 333 us; skew 500 ppm
I20260619 13:03:20.483335 14601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:20 dist-test-slave-tpsm krb5kdc[13245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:20.489785 14601 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:03:20.535115 14601 webserver.cc:460] Webserver started at http://127.1.87.67:41025/ using document root <none> and password file <none>
I20260619 13:03:20.536412 14601 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:20.536674 14601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:03:20 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:20.581539 14601 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.007s sys 0.005s
I20260619 13:03:20.589236 14619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:20.591085 14601 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260619 13:03:20.591272 14601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "ced20bed01cd4bf583da68c21547aca1"
format_stamp: "Formatted at 2026-06-19 13:03:17 on dist-test-slave-tpsm"
server_key: "1730b3f50ca3028d49a1445975af8cce"
server_key_iv: "f661bc9dda8a2fca01d47e9e0fe48945"
server_key_version: "kuduclusterkey@0"
I20260619 13:03:20.592574 14601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:20.646522 14601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:20.651816 14601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:20.652159 14601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:20.653335 14601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:20.655027 14601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:20.655138 14601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:20.655238 14601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:20.655331 14601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:20.703653 14601 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:45815
I20260619 13:03:20.703737 14731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:45815 every 8 connection(s)
I20260619 13:03:20.705453 14601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:03:20.713914 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 14601
Jun 19 13:03:20 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:20.748734 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.715515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.1.87.126:45513)
0619 13:03:20.730810 (+ 15295us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:20.730816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:20.730837 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:03:20.730886 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:20.730891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:20.730984 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:20.731102 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:20.733157 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.734404 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.736256 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.736870 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.736983 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.740830 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:20.740836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:20.740840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:20.740886 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:20.744702 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:20.745660 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:20.745665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:20.745668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:20.745806 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:20.746247 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:20.746252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:20.746255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:20.746568 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:20.746821 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:20.748188 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0619 13:03:20.748401 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15159,"thread_start_us":9713,"threads_started":1}
I20260619 13:03:20.749023 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.713020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45513 (local address 127.1.87.67:41605)
0619 13:03:20.720065 (+ 7045us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:20.720145 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:20.724570 (+ 4425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:20.726490 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:20.726510 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:20.727363 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:20.728409 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:20.728438 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.730482 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.730490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:20.732452 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:20.732479 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.733664 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.733668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:20.733870 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.735547 (+ 1677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:20.735578 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:20.738321 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:20.743776 (+ 5455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:20.743788 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:20.743811 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:20.744490 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:20.744963 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:20.744968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:20.744972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:20.745227 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:20.745972 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:20.745981 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:20.746430 (+ 449us) client_negotiation.cc:769] Sending connection context
0619 13:03:20.746772 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 13:03:20.747387 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6638,"thread_start_us":6343,"threads_started":1}
I20260619 13:03:20.750138 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.714837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:41605 (local address 127.1.87.126:45513)
0619 13:03:20.725519 (+ 10682us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:20.725534 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:20.725559 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:03:20.725637 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:20.725645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:20.725771 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:20.725905 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:20.728670 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.730209 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.732641 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.733266 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.733348 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.738575 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:20.738581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:20.738587 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:20.738624 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:20.743576 (+ 4952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:20.744655 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:20.744660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:20.744663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:20.744819 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:20.745370 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:20.745375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:20.745378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:20.745753 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:20.745892 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:20.749535 (+ 3643us) server_negotiation.cc:299] Negotiation successful
0619 13:03:20.749754 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10515,"thread_start_us":15687,"threads_started":1}
I20260619 13:03:20.752069 14732 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45513
I20260619 13:03:20.752420 14732 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:20.753342 14732 heartbeater.cc:511] Master 127.1.87.126:45513 requested a full tablet report, sending...
I20260619 13:03:20.755704 14256 ts_manager.cc:194] Registered new tserver with Master: ced20bed01cd4bf583da68c21547aca1 (127.1.87.67:45815)
I20260619 13:03:20.756850 14256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:41605
I20260619 13:03:20.767755 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:03:20 dist-test-slave-tpsm krb5kdc[13245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for HTTP/127.1.87.84@KRBTEST.COM
I20260619 13:03:20.814126 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.783920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.1.87.126:45513)
0619 13:03:20.784131 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:20.784137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:20.784227 (+ 90us) server_negotiation.cc:407] Connection header received
0619 13:03:20.784480 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:20.784486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:20.784562 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:20.784630 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:20.792833 (+ 8203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.793960 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.802871 (+ 8911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.803435 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.803572 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.807299 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:20.807307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:20.807335 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:20.807385 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:20.810812 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:20.811703 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:20.811709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:20.811714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:20.811865 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:20.812332 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:20.812338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:20.812341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:20.812715 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:20.812955 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:20.813538 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 13:03:20.813780 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:03:20.821115 14256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:03:20.832603 14256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:03:20.956001 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.887502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.1.87.67:45815)
0619 13:03:20.891976 (+ 4474us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:20.891990 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:20.944533 (+ 52543us) server_negotiation.cc:407] Connection header received
0619 13:03:20.944864 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:20.944874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:20.945164 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:20.945350 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:20.947825 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.949744 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.952876 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.954345 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.954450 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.954689 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:20.955462 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 13:03:20.955645 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4245,"thread_start_us":4012,"threads_started":1}
I20260619 13:03:20.957113 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.887119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:45815 (local address 127.0.0.1:47166)
0619 13:03:20.944343 (+ 57224us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:20.944372 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:20.944576 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:20.947056 (+ 2480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:20.947064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:20.947101 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:20.947576 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:20.947591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.949982 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.949988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:20.952673 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:20.952690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.954898 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.954903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:20.955114 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.955185 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 13:03:20.956535 (+ 1350us) client_negotiation.cc:240] Negotiation successful
0619 13:03:20.956757 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57080,"thread_start_us":57886,"threads_started":1}
I20260619 13:03:20.961575 14667 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba5eb8118914086b34a00b69224f994 (DEFAULT_TABLE table=test-table [id=83af998de0e14b64ac5dad74f83102e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:20.963451 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba5eb8118914086b34a00b69224f994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:20.969856 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.885258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.1.87.65:45991)
0619 13:03:20.907366 (+ 22108us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:20.907376 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:20.938693 (+ 31317us) server_negotiation.cc:407] Connection header received
0619 13:03:20.949247 (+ 10554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:20.949256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:20.949472 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:20.949616 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:20.953244 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.954787 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.963275 (+ 8488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.964818 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.964933 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.965149 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:20.969317 (+ 4168us) server_negotiation.cc:299] Negotiation successful
0619 13:03:20.969510 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21874,"thread_start_us":22117,"threads_started":1}
I20260619 13:03:20.967969 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.895554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:43527 (local address 127.0.0.1:55504)
0619 13:03:20.947860 (+ 52306us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:20.947881 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:20.948006 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:20.950712 (+ 2706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:20.950717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:20.950743 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:20.951142 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:20.951156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.957267 (+ 6111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.957272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:20.959259 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:20.959275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.967083 (+ 7808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.967089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:20.967283 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.967336 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 13:03:20.967463 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 13:03:20.967615 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52176,"thread_start_us":50202,"threads_started":1}
I20260619 13:03:20.970611 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.884816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:45991 (local address 127.0.0.1:46558)
0619 13:03:20.938461 (+ 53645us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:20.938526 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:20.938788 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:20.949883 (+ 11095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:20.949888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:20.949916 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:20.950431 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:20.950445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.954975 (+ 4530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.954980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:20.957091 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:20.957110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.966646 (+ 9536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.966652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:20.966842 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.966878 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:03:20.970123 (+ 3245us) client_negotiation.cc:240] Negotiation successful
0619 13:03:20.970293 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53299,"thread_start_us":64015,"threads_started":1}
I20260619 13:03:20.973739 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:20.896138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.1.87.66:43527)
0619 13:03:20.943906 (+ 47768us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:20.943916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:20.948129 (+ 4213us) server_negotiation.cc:407] Connection header received
0619 13:03:20.948222 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:20.948242 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:20.948487 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:20.948673 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:20.951388 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.953020 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.959485 (+ 6465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:20.961014 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:20.961143 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:20.961387 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:20.970756 (+ 9369us) server_negotiation.cc:299] Negotiation successful
0619 13:03:20.973405 (+ 2649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47550,"thread_start_us":48682,"threads_started":1}
I20260619 13:03:20.975373 14388 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba5eb8118914086b34a00b69224f994 (DEFAULT_TABLE table=test-table [id=83af998de0e14b64ac5dad74f83102e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:20.977478 14531 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba5eb8118914086b34a00b69224f994 (DEFAULT_TABLE table=test-table [id=83af998de0e14b64ac5dad74f83102e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:20.978891 14531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba5eb8118914086b34a00b69224f994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:20.980810 14388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba5eb8118914086b34a00b69224f994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:21.004418 14755 tablet_bootstrap.cc:492] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: Bootstrap starting.
I20260619 13:03:21.005645 14598 heartbeater.cc:503] Master 127.1.87.126:45513 was elected leader, sending a full tablet report...
I20260619 13:03:21.007961 14756 tablet_bootstrap.cc:492] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: Bootstrap starting.
I20260619 13:03:21.015817 14755 tablet_bootstrap.cc:654] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:21.016539 14756 tablet_bootstrap.cc:654] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:21.017294 14755 log.cc:824] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:21.018141 14756 log.cc:824] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:21.019232 14757 tablet_bootstrap.cc:492] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: Bootstrap starting.
I20260619 13:03:21.023926 14756 tablet_bootstrap.cc:492] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: No bootstrap required, opened a new log
I20260619 13:03:21.024318 14756 ts_tablet_manager.cc:1397] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.002s
I20260619 13:03:21.025866 14757 tablet_bootstrap.cc:654] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:21.031992 14757 log.cc:824] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:21.036114 14755 tablet_bootstrap.cc:492] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: No bootstrap required, opened a new log
I20260619 13:03:21.036455 14755 ts_tablet_manager.cc:1397] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.004s
I20260619 13:03:21.047176 14756 raft_consensus.cc:348] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.047891 14756 raft_consensus.cc:374] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:21.048043 14756 raft_consensus.cc:729] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667d939964c449a98ed79c582d34e5f7, State: Initialized, Role: FOLLOWER
I20260619 13:03:21.050552 14756 consensus_queue.cc:260] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.052168 14757 tablet_bootstrap.cc:492] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: No bootstrap required, opened a new log
I20260619 13:03:21.052522 14757 ts_tablet_manager.cc:1397] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.009s
I20260619 13:03:21.058952 14756 ts_tablet_manager.cc:1428] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: Time spent starting tablet: real 0.034s user 0.017s sys 0.002s
I20260619 13:03:21.060639 14755 raft_consensus.cc:348] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.063457 14755 raft_consensus.cc:374] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:21.063627 14755 raft_consensus.cc:729] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced20bed01cd4bf583da68c21547aca1, State: Initialized, Role: FOLLOWER
I20260619 13:03:21.064465 14755 consensus_queue.cc:260] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.067180 14757 raft_consensus.cc:348] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.067766 14757 raft_consensus.cc:374] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:21.067910 14757 raft_consensus.cc:729] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2872e3dfd774891873632d7bffd8d6a, State: Initialized, Role: FOLLOWER
I20260619 13:03:21.068759 14757 consensus_queue.cc:260] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.072988 14454 heartbeater.cc:503] Master 127.1.87.126:45513 was elected leader, sending a full tablet report...
I20260619 13:03:21.073474 14732 heartbeater.cc:503] Master 127.1.87.126:45513 was elected leader, sending a full tablet report...
I20260619 13:03:21.074733 14755 ts_tablet_manager.cc:1428] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: Time spent starting tablet: real 0.038s user 0.016s sys 0.000s
I20260619 13:03:21.075817 14757 ts_tablet_manager.cc:1428] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: Time spent starting tablet: real 0.023s user 0.012s sys 0.002s
W20260619 13:03:21.126873 14455 tablet.cc:2369] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:03:21.214725 14733 tablet.cc:2369] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:03:21.231069 14602 tablet.cc:2369] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:03:21.293283 14765 raft_consensus.cc:482] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:03:21.293632 14765 raft_consensus.cc:504] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.295214 14763 raft_consensus.cc:482] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:03:21.295454 14765 leader_election.cc:288] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced20bed01cd4bf583da68c21547aca1 (127.1.87.67:45815), 667d939964c449a98ed79c582d34e5f7 (127.1.87.66:43527)
I20260619 13:03:21.295495 14763 raft_consensus.cc:504] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.297210 14763 leader_election.cc:288] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2872e3dfd774891873632d7bffd8d6a (127.1.87.65:45991), 667d939964c449a98ed79c582d34e5f7 (127.1.87.66:43527)
I20260619 13:03:21.305166 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.295880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:43527 (local address 127.1.87.65:41515)
0619 13:03:21.296999 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:21.297034 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:21.297241 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:21.297813 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:21.297818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:21.297855 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:21.298397 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:21.298409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.300092 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.300099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:21.302538 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:21.302552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.304306 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.304313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:21.304485 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.304520 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:03:21.304624 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 13:03:21.304765 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":84,"thread_start_us":482,"threads_started":1}
I20260619 13:03:21.305603 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.296477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:41515 (local address 127.1.87.66:43527)
0619 13:03:21.296705 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:21.296713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:21.297131 (+ 418us) server_negotiation.cc:407] Connection header received
0619 13:03:21.297406 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:21.297415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:21.297517 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:21.297591 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:21.298667 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.299938 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.302693 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.303936 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.304053 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.304156 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:21.305193 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0619 13:03:21.305347 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:03:21.306295 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.297780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:53335 (local address 127.1.87.65:45991)
0619 13:03:21.297958 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:21.297965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:21.298639 (+ 674us) server_negotiation.cc:407] Connection header received
0619 13:03:21.298871 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:21.298877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:21.298945 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:21.299017 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:21.299864 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.301050 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.303389 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.304628 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.304742 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.304822 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:21.305934 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0619 13:03:21.306059 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":43}
I20260619 13:03:21.306412 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.297312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:45991 (local address 127.1.87.67:53335)
0619 13:03:21.298540 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:21.298560 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:21.298727 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:21.299307 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:21.299339 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:21.299368 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:21.299682 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:21.299700 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.301244 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.301248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:21.303179 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:21.303192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.304908 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.304917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:21.305770 (+ 853us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.305804 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:03:21.305888 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 13:03:21.306023 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":627,"thread_start_us":1002,"threads_started":1}
I20260619 13:03:21.306690 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba5eb8118914086b34a00b69224f994" candidate_uuid: "f2872e3dfd774891873632d7bffd8d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667d939964c449a98ed79c582d34e5f7" is_pre_election: true
I20260619 13:03:21.307192 14551 raft_consensus.cc:2454] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2872e3dfd774891873632d7bffd8d6a in term 0.
I20260619 13:03:21.307247 14408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba5eb8118914086b34a00b69224f994" candidate_uuid: "ced20bed01cd4bf583da68c21547aca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2872e3dfd774891873632d7bffd8d6a" is_pre_election: true
I20260619 13:03:21.307765 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.298069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:56217 (local address 127.1.87.66:43527)
0619 13:03:21.298922 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:21.298929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:21.299748 (+ 819us) server_negotiation.cc:407] Connection header received
0619 13:03:21.299986 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:21.299991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:21.300057 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:21.300122 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:21.301053 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.302287 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.305138 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.306532 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.306647 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.306786 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:21.307287 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 13:03:21.307436 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":521,"threads_started":1}
I20260619 13:03:21.307662 14408 raft_consensus.cc:2454] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced20bed01cd4bf583da68c21547aca1 in term 0.
I20260619 13:03:21.307799 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.297656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:43527 (local address 127.1.87.67:56217)
0619 13:03:21.299655 (+ 1999us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:21.299677 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:21.299834 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:21.300400 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:21.300404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:21.300427 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:21.300833 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:21.300843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.302716 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.302724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:21.304960 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:21.304979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.306918 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.306924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:21.307111 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.307150 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 13:03:21.307262 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 13:03:21.307422 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1810,"mutex_wait_us":84,"thread_start_us":1613,"threads_started":1}
I20260619 13:03:21.308012 14342 leader_election.cc:302] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667d939964c449a98ed79c582d34e5f7, f2872e3dfd774891873632d7bffd8d6a; no voters:
I20260619 13:03:21.308369 14623 leader_election.cc:302] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced20bed01cd4bf583da68c21547aca1, f2872e3dfd774891873632d7bffd8d6a; no voters:
I20260619 13:03:21.308427 14765 raft_consensus.cc:2781] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:03:21.308568 14765 raft_consensus.cc:482] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:03:21.308681 14765 raft_consensus.cc:3037] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:21.308748 14763 raft_consensus.cc:2781] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:03:21.308728 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba5eb8118914086b34a00b69224f994" candidate_uuid: "ced20bed01cd4bf583da68c21547aca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667d939964c449a98ed79c582d34e5f7" is_pre_election: true
I20260619 13:03:21.308881 14763 raft_consensus.cc:482] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:03:21.308905 14551 raft_consensus.cc:2454] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced20bed01cd4bf583da68c21547aca1 in term 0.
I20260619 13:03:21.308974 14763 raft_consensus.cc:3037] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:21.309055 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.295775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:45815 (local address 127.1.87.65:59669)
0619 13:03:21.296701 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:21.296786 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:21.297003 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:21.297632 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:21.297636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:21.297666 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:21.299629 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:21.299642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.301176 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.301183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:21.303506 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:21.303519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.305418 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.305424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:21.305602 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.305638 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:03:21.308112 (+ 2474us) client_negotiation.cc:240] Negotiation successful
0619 13:03:21.308268 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":533,"thread_start_us":935,"threads_started":1}
I20260619 13:03:21.309439 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.296021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:59669 (local address 127.1.87.67:45815)
0619 13:03:21.296195 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:21.296202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:21.296882 (+ 680us) server_negotiation.cc:407] Connection header received
0619 13:03:21.297183 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:21.297194 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:21.297272 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:21.297341 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:21.299780 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.300960 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.303691 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.304990 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.305161 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.305265 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:21.308573 (+ 3308us) server_negotiation.cc:299] Negotiation successful
0619 13:03:21.308697 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":59}
I20260619 13:03:21.310356 14687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba5eb8118914086b34a00b69224f994" candidate_uuid: "f2872e3dfd774891873632d7bffd8d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced20bed01cd4bf583da68c21547aca1" is_pre_election: true
I20260619 13:03:21.311555 14763 raft_consensus.cc:504] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.311769 14765 raft_consensus.cc:504] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.312014 14687 raft_consensus.cc:2379] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2872e3dfd774891873632d7bffd8d6a in current term 1: Already voted for candidate ced20bed01cd4bf583da68c21547aca1 in this term.
I20260619 13:03:21.312582 14408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba5eb8118914086b34a00b69224f994" candidate_uuid: "ced20bed01cd4bf583da68c21547aca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2872e3dfd774891873632d7bffd8d6a"
I20260619 13:03:21.312580 14551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba5eb8118914086b34a00b69224f994" candidate_uuid: "ced20bed01cd4bf583da68c21547aca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667d939964c449a98ed79c582d34e5f7"
I20260619 13:03:21.312769 14551 raft_consensus.cc:3037] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:21.312866 14408 raft_consensus.cc:2379] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ced20bed01cd4bf583da68c21547aca1 in current term 1: Already voted for candidate f2872e3dfd774891873632d7bffd8d6a in this term.
I20260619 13:03:21.312877 14765 leader_election.cc:288] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [CANDIDATE]: Term 1 election: Requested vote from peers ced20bed01cd4bf583da68c21547aca1 (127.1.87.67:45815), 667d939964c449a98ed79c582d34e5f7 (127.1.87.66:43527)
I20260619 13:03:21.313426 14686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba5eb8118914086b34a00b69224f994" candidate_uuid: "f2872e3dfd774891873632d7bffd8d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced20bed01cd4bf583da68c21547aca1"
I20260619 13:03:21.313740 14686 raft_consensus.cc:2379] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f2872e3dfd774891873632d7bffd8d6a in current term 1: Already voted for candidate ced20bed01cd4bf583da68c21547aca1 in this term.
I20260619 13:03:21.313859 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba5eb8118914086b34a00b69224f994" candidate_uuid: "f2872e3dfd774891873632d7bffd8d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667d939964c449a98ed79c582d34e5f7"
I20260619 13:03:21.314169 14763 leader_election.cc:288] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [CANDIDATE]: Term 1 election: Requested vote from peers f2872e3dfd774891873632d7bffd8d6a (127.1.87.65:45991), 667d939964c449a98ed79c582d34e5f7 (127.1.87.66:43527)
I20260619 13:03:21.314648 14342 leader_election.cc:302] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2872e3dfd774891873632d7bffd8d6a; no voters: 667d939964c449a98ed79c582d34e5f7, ced20bed01cd4bf583da68c21547aca1
I20260619 13:03:21.314908 14765 raft_consensus.cc:2726] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 13:03:21.315888 14551 raft_consensus.cc:2454] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced20bed01cd4bf583da68c21547aca1 in term 1.
I20260619 13:03:21.316318 14621 leader_election.cc:302] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 667d939964c449a98ed79c582d34e5f7, ced20bed01cd4bf583da68c21547aca1; no voters: f2872e3dfd774891873632d7bffd8d6a
I20260619 13:03:21.316656 14763 raft_consensus.cc:2781] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:03:21.317180 14763 raft_consensus.cc:686] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 1 LEADER]: Becoming Leader. State: Replica: ced20bed01cd4bf583da68c21547aca1, State: Running, Role: LEADER
I20260619 13:03:21.317692 14763 consensus_queue.cc:237] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } }
I20260619 13:03:21.321951 14256 catalog_manager.cc:5495] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced20bed01cd4bf583da68c21547aca1 (127.1.87.67). New cstate: current_term: 1 leader_uuid: "ced20bed01cd4bf583da68c21547aca1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced20bed01cd4bf583da68c21547aca1" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 45815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 } health_report { overall_health: UNKNOWN } } }
I20260619 13:03:21.347067 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.341424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.1.87.67:45815)
0619 13:03:21.341586 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:21.341590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:21.342091 (+ 501us) server_negotiation.cc:407] Connection header received
0619 13:03:21.342275 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:21.342279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:21.342372 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:21.342488 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:03:21.342858 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.343646 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.345116 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.345517 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.345609 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.346065 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:03:21.346546 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:03:21.346649 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:21.346786 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 13:03:21.346846 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:03:21.355052 14551 raft_consensus.cc:1261] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 1 FOLLOWER]: Refusing update from remote peer ced20bed01cd4bf583da68c21547aca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:21.355160 14408 raft_consensus.cc:1261] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 1 FOLLOWER]: Refusing update from remote peer ced20bed01cd4bf583da68c21547aca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:21.355955 14776 consensus_queue.cc:1035] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "667d939964c449a98ed79c582d34e5f7" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:03:21.356276 14763 consensus_queue.cc:1035] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2872e3dfd774891873632d7bffd8d6a" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 45991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:03:21.375801 14780 mvcc.cc:205] Tried to move back new op lower bound from 7298556728741347328 to 7298556728600637440. Current Snapshot: MvccSnapshot[applied={T|T < 7298556728741347328}]
I20260619 13:03:21.382314 14782 mvcc.cc:205] Tried to move back new op lower bound from 7298556728741347328 to 7298556728600637440. Current Snapshot: MvccSnapshot[applied={T|T < 7298556728741347328}]
I20260619 13:03:21.392009 14256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:03:21.392285 14256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:03:21.395701 14256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d2ef446470e4225874237487c775bd1: Sending DeleteTablet for 3 replicas of tablet 6ba5eb8118914086b34a00b69224f994
I20260619 13:03:21.396844 14531 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba5eb8118914086b34a00b69224f994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:21 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:55504
I20260619 13:03:21.396847 14667 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba5eb8118914086b34a00b69224f994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:21 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:47166
I20260619 13:03:21.396839 14388 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba5eb8118914086b34a00b69224f994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:21 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:46558
I20260619 13:03:21.399485 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 14293
I20260619 13:03:21.409857 14790 tablet_replica.cc:330] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: stopping tablet replica
I20260619 13:03:21.410487 14791 tablet_replica.cc:330] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: stopping tablet replica
I20260619 13:03:21.411167 14791 raft_consensus.cc:2229] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:03:21.411695 14790 raft_consensus.cc:2229] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:03:21.412294 14790 raft_consensus.cc:2258] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:21.412778 14791 raft_consensus.cc:2258] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:21.414124 14792 tablet_replica.cc:330] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: stopping tablet replica
I20260619 13:03:21.414487 14792 raft_consensus.cc:2229] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:03:21.415105 14792 raft_consensus.cc:2258] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:21.416798 14790 ts_tablet_manager.cc:1905] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:21.417218 14791 ts_tablet_manager.cc:1905] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:21.424315 14791 ts_tablet_manager.cc:1918] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:21.424335 14790 ts_tablet_manager.cc:1918] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:21.424481 14791 log.cc:1196] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/6ba5eb8118914086b34a00b69224f994
I20260619 13:03:21.424666 14790 log.cc:1196] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/6ba5eb8118914086b34a00b69224f994
I20260619 13:03:21.425021 14791 ts_tablet_manager.cc:1939] T 6ba5eb8118914086b34a00b69224f994 P 667d939964c449a98ed79c582d34e5f7: Deleting consensus metadata
I20260619 13:03:21.425343 14790 ts_tablet_manager.cc:1939] T 6ba5eb8118914086b34a00b69224f994 P f2872e3dfd774891873632d7bffd8d6a: Deleting consensus metadata
I20260619 13:03:21.426990 14241 catalog_manager.cc:4841] TS 667d939964c449a98ed79c582d34e5f7 (127.1.87.66:43527): tablet 6ba5eb8118914086b34a00b69224f994 (table test-table [id=83af998de0e14b64ac5dad74f83102e1]) successfully deleted
I20260619 13:03:21.427394 14243 catalog_manager.cc:4841] TS f2872e3dfd774891873632d7bffd8d6a (127.1.87.65:45991): tablet 6ba5eb8118914086b34a00b69224f994 (table test-table [id=83af998de0e14b64ac5dad74f83102e1]) successfully deleted
I20260619 13:03:21.428711 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.401074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.1.87.65:45991)
0619 13:03:21.401300 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:21.401310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:21.405456 (+ 4146us) server_negotiation.cc:407] Connection header received
0619 13:03:21.405649 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:21.405658 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:21.405768 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:21.405896 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:21.412568 (+ 6672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.414029 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.415901 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.416545 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.416688 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.420178 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:21.420191 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:21.420207 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:21.420247 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:21.424438 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:21.425410 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:21.425419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:21.425422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:21.425571 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:21.426059 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:21.426064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:21.426067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:21.426516 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:21.427794 (+ 1278us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:21.428128 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 13:03:21.428347 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:03:21.428673 14792 ts_tablet_manager.cc:1905] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:21.429396 14449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:21.436719 14792 ts_tablet_manager.cc:1918] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:21.436894 14792 log.cc:1196] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/6ba5eb8118914086b34a00b69224f994
I20260619 13:03:21.437484 14792 ts_tablet_manager.cc:1939] T 6ba5eb8118914086b34a00b69224f994 P ced20bed01cd4bf583da68c21547aca1: Deleting consensus metadata
I20260619 13:03:21.439388 14240 catalog_manager.cc:4841] TS ced20bed01cd4bf583da68c21547aca1 (127.1.87.67:45815): tablet 6ba5eb8118914086b34a00b69224f994 (table test-table [id=83af998de0e14b64ac5dad74f83102e1]) successfully deleted
I20260619 13:03:21.524524 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 14293
I20260619 13:03:21.542711 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 14458
I20260619 13:03:21.560297 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.543571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.1.87.66:43527)
0619 13:03:21.543763 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:21.543768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:21.543809 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:03:21.544040 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:21.544044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:21.544145 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:21.544246 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:21.545926 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.547046 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.549111 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.549605 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.549783 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.553155 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:21.553178 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:21.553201 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:21.553255 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:21.556716 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:21.557650 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:21.557658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:21.557661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:21.557839 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:21.558275 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:21.558280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:21.558282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:21.558762 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:21.559008 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:21.559507 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 13:03:21.559816 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 13:03:21.560994 14591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:21.653859 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 14458
I20260619 13:03:21.671380 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 14601
I20260619 13:03:21.687965 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:21.672402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.1.87.67:45815)
0619 13:03:21.672610 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:21.672614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:21.672631 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:03:21.672685 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:21.672688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:21.672744 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:21.672825 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:21.674494 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.675248 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.677192 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:21.677762 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:21.677871 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:21.681251 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:21.681262 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:21.681279 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:21.681331 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:21.684965 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:21.685850 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:21.685857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:21.685859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:21.685998 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:21.686524 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:21.686528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:21.686530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:21.686894 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:21.687143 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:21.687543 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 13:03:21.687721 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:03:21.688807 14727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:21.810694 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 14601
I20260619 13:03:21.829895 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 14223
I20260619 13:03:21.831596 14286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:21.943799 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 14223
I20260619 13:03:21.968228 1373 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 13:03:22.303887 1373 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 13:03:22.304103 1373 mini_ranger.cc:66] Stopping Ranger...
I20260619 13:03:22.649966 1373 mini_ranger.cc:68] Stopped Ranger
2026-06-19 13:03:22.650 UTC [13275] LOG: received smart shutdown request
2026-06-19 13:03:22.652 UTC [13275] LOG: background worker "logical replication launcher" (PID 13284) exited with exit code 1
2026-06-19 13:03:22.653 UTC [13279] LOG: shutting down
2026-06-19 13:03:22.787 UTC [13275] LOG: database system is shut down
2026-06-19T13:03:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51517 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:03:23 dist-test-slave-tpsm krb5kdc[14806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:03:23 dist-test-slave-tpsm krb5kdc[14806](info): set up 2 sockets
Jun 19 13:03:23 dist-test-slave-tpsm krb5kdc[14806](info): commencing operation
krb5kdc: starting...
W20260619 13:03:24.047747 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:03:24 dist-test-slave-tpsm krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:03:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:03:24.268630 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39531
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:39531,127.1.87.125:37003,127.1.87.124:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:24.599089 14819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:24.599452 14819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:24.599635 14819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:24.606917 14819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:24.607036 14819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:24.607153 14819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:24.607271 14819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:03:24.607393 14819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:24.617652 14819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:39531,127.1.87.125:37003,127.1.87.124:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:24.619059 14819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:24.620445 14819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:24.632607 14824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:24.632684 14825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:24.632652 14827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:24.633502 14819 server_base.cc:1034] running on GCE node
I20260619 13:03:24.634255 14819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:24.635690 14819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:24.636910 14819 hybrid_clock.cc:648] HybridClock initialized: now 1781874204636870 us; error 37 us; skew 500 ppm
I20260619 13:03:24.637251 14819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:24 dist-test-slave-tpsm krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:24.643736 14819 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:03:24.645411 14819 webserver.cc:460] Webserver started at http://127.1.87.126:45545/ using document root <none> and password file <none>
I20260619 13:03:24.646190 14819 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:24.646350 14819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:24.646708 14819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:24.649994 14819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "f1944b05a7d84831abe21acd8a844ff2"
format_stamp: "Formatted at 2026-06-19 13:03:24 on dist-test-slave-tpsm"
server_key: "2e262ecb0b66680c79841365262eed14"
server_key_iv: "decc7d0f58dfa341fd2b25471e1710af"
server_key_version: "encryptionkey@0"
I20260619 13:03:24.650910 14819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "f1944b05a7d84831abe21acd8a844ff2"
format_stamp: "Formatted at 2026-06-19 13:03:24 on dist-test-slave-tpsm"
server_key: "2e262ecb0b66680c79841365262eed14"
server_key_iv: "decc7d0f58dfa341fd2b25471e1710af"
server_key_version: "encryptionkey@0"
I20260619 13:03:24.656626 14819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:03:24.660226 14834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:24.661710 14819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 13:03:24.661918 14819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "f1944b05a7d84831abe21acd8a844ff2"
format_stamp: "Formatted at 2026-06-19 13:03:24 on dist-test-slave-tpsm"
server_key: "2e262ecb0b66680c79841365262eed14"
server_key_iv: "decc7d0f58dfa341fd2b25471e1710af"
server_key_version: "encryptionkey@0"
I20260619 13:03:24.662148 14819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:24.684648 14819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:24.689159 14819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:24.689498 14819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:24.710361 14819 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:39531
I20260619 13:03:24.710366 14885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:39531 every 8 connection(s)
I20260619 13:03:24.711762 14819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:03:24.715409 14886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:24.720928 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 14819
I20260619 13:03:24.721195 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:03:24.721858 1373 external_mini_cluster.cc:1428] Setting key 040c04e1214c422653ae394f0c04c73e
I20260619 13:03:24.723963 14886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:03:24 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.87.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.125@KRBTEST.COM" created.
I20260619 13:03:24.749307 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.727497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39531 (local address 127.0.0.1:58836)
0619 13:03:24.728435 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:24.728492 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:24.731169 (+ 2677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:24.732127 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:24.732133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:24.732675 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:24.733013 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:24.733030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:24.735640 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:24.735645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:24.736782 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:24.736795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:24.738195 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:24.738200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:24.738342 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:24.739075 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:24.739105 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:24.741522 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:24.744813 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:24.744824 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:24.744836 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:24.745229 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:24.745666 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:24.745673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:24.745677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:24.745824 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:24.746648 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:24.746659 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:24.746918 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 13:03:24.747214 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 13:03:24.747910 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":213,"thread_start_us":413,"threads_started":1}
I20260619 13:03:24.749327 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.728809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.1.87.126:39531)
0619 13:03:24.729485 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:24.729496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:24.729549 (+ 53us) server_negotiation.cc:407] Connection header received
0619 13:03:24.731576 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:24.731586 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:24.731885 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:24.732104 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:24.733216 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:24.735448 (+ 2232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:24.736949 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:24.737939 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:24.738162 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:24.741771 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:24.741780 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:24.741793 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:24.741821 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:24.744668 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:24.745360 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:24.745376 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:24.745378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:24.745549 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:24.745929 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:24.745936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:24.745940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:24.746508 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:24.746794 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:24.747360 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 13:03:24.747809 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":488,"thread_start_us":331,"threads_started":1}
I20260619 13:03:24.752301 14886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:03:24.753443 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.753040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50474)
0619 13:03:24.753221 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:24.753328 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 13:03:24.753894 14835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
I20260619 13:03:24.755530 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.755135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50480)
0619 13:03:24.755273 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:24.755402 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:03:24.755824 14886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:37003: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab.
Entry for principal kudu/127.1.87.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab.
I20260619 13:03:24.772647 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.125:37003
--webserver_interface=127.1.87.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab
--principal=kudu/127.1.87.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:39531,127.1.87.125:37003,127.1.87.124:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:03:24.805186 14886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } attempt: 1
I20260619 13:03:24.806309 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.805872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50492)
0619 13:03:24.806071 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:24.806180 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 13:03:24.807302 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.806979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50498)
0619 13:03:24.807135 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:24.807195 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:03:24.807637 14886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:37003: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
I20260619 13:03:24.875911 14886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } attempt: 2
I20260619 13:03:24.877127 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.876632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50504)
0619 13:03:24.876891 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:24.876976 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 13:03:24.878126 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.877775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50516)
0619 13:03:24.877929 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:24.878002 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:03:24.878368 14886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:37003: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
I20260619 13:03:24.969710 14886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } attempt: 3
I20260619 13:03:24.971516 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.970745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50532)
0619 13:03:24.971069 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:24.971276 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260619 13:03:24.972769 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:24.972371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50536)
0619 13:03:24.972536 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:24.972608 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 13:03:24.973042 14886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:37003: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
W20260619 13:03:25.108512 14893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:25.108811 14893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:25.108902 14893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:25.116138 14893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:25.116281 14893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:25.116365 14893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:25.116460 14893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260619 13:03:25.116358 14886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } attempt: 4
W20260619 13:03:25.116526 14893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:25.117565 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.117023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50546)
0619 13:03:25.117340 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.117432 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260619 13:03:25.118547 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.118190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50552)
0619 13:03:25.118333 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.118429 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:03:25.118737 14886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:37003: Network error: Client connection negotiation failed: client connection to 127.1.87.125:37003: connect: Connection refused (error 111)
I20260619 13:03:25.127048 14893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:39531,127.1.87.125:37003,127.1.87.124:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab
--principal=kudu/127.1.87.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.125:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.87.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:25.128504 14893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:25.129895 14893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:25.142246 14898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:25.142169 14899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:25.142204 14901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:25.142976 14893 server_base.cc:1034] running on GCE node
I20260619 13:03:25.143807 14893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:25.145238 14893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:25.146524 14893 hybrid_clock.cc:648] HybridClock initialized: now 1781874205146480 us; error 33 us; skew 500 ppm
I20260619 13:03:25.146943 14893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874205, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:25.150465 14893 init.cc:377] Logged in from keytab as kudu/127.1.87.125@KRBTEST.COM (short username kudu)
I20260619 13:03:25.152006 14893 webserver.cc:460] Webserver started at http://127.1.87.125:43651/ using document root <none> and password file <none>
I20260619 13:03:25.152679 14893 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:25.152783 14893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:25.153024 14893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:25.156337 14893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/data/instance:
uuid: "1fcc23a96b824502845a7c05d60657f9"
format_stamp: "Formatted at 2026-06-19 13:03:25 on dist-test-slave-tpsm"
server_key: "0cf0e21bf78ab87219761a8ed5d7c8cf"
server_key_iv: "24c101ebe374034668f7c2e3ed1f8a86"
server_key_version: "encryptionkey@0"
I20260619 13:03:25.157101 14893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/wal/instance:
uuid: "1fcc23a96b824502845a7c05d60657f9"
format_stamp: "Formatted at 2026-06-19 13:03:25 on dist-test-slave-tpsm"
server_key: "0cf0e21bf78ab87219761a8ed5d7c8cf"
server_key_iv: "24c101ebe374034668f7c2e3ed1f8a86"
server_key_version: "encryptionkey@0"
I20260619 13:03:25.162638 14893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:03:25.166489 14908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:25.167987 14893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:25.168131 14893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/wal
uuid: "1fcc23a96b824502845a7c05d60657f9"
format_stamp: "Formatted at 2026-06-19 13:03:25 on dist-test-slave-tpsm"
server_key: "0cf0e21bf78ab87219761a8ed5d7c8cf"
server_key_iv: "24c101ebe374034668f7c2e3ed1f8a86"
server_key_version: "encryptionkey@0"
I20260619 13:03:25.168303 14893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:25.185286 14893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:25.189638 14893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:25.189869 14893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:25.209359 14893 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.125:37003
I20260619 13:03:25.209391 14959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.125:37003 every 8 connection(s)
I20260619 13:03:25.210737 14893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
I20260619 13:03:25.214391 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:25.215775 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 14893
I20260619 13:03:25.215971 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-1/wal/instance
I20260619 13:03:25.216548 1373 external_mini_cluster.cc:1428] Setting key 26dac831dda09258335c30a4fffde2e5
I20260619 13:03:25.224547 14960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.87.124@KRBTEST.COM; defaulting to no policy
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874205, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.124@KRBTEST.COM" created.
I20260619 13:03:25.248118 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.227484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.1.87.126:39531)
0619 13:03:25.227670 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.227675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.229183 (+ 1508us) server_negotiation.cc:407] Connection header received
0619 13:03:25.231129 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.231134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.231225 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.231360 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.232850 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.233529 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.235184 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.235690 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.235828 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.240320 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.240326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.240331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.240372 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.244319 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.245174 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.245183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.245189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.245344 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.245841 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.245850 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.245856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.246318 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.246473 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.247263 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.247579 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:03:25.249094 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.228146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39531 (local address 127.0.0.1:58840)
0619 13:03:25.229024 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.229061 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:25.230848 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:25.231596 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:25.231605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:25.232118 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:25.232678 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.232694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.233706 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.233710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.235027 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.235038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.235922 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.235929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.236147 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.236922 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:25.236943 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:25.240090 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:25.244512 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.244524 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.244545 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.245011 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.245497 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.245501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.245503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.245702 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.246465 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:25.246471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:25.246861 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 13:03:25.247095 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 13:03:25.247622 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":403,"threads_started":1}
I20260619 13:03:25.251308 14960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874205, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
I20260619 13:03:25.266961 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.252016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50568)
0619 13:03:25.252208 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.252227 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:25.252333 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:25.253311 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:25.253318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:25.253831 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:25.254043 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.254056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.255410 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.255418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.256974 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.256991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.257796 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.257804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.258066 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.258669 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:25.258697 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:25.261317 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:25.264486 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.264493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.264496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.264831 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.265205 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.265212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.265218 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.265356 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.265938 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:25.265946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:25.266078 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 13:03:25.266263 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 13:03:25.266557 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 13:03:25.267129 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.252241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.1.87.125:37003)
0619 13:03:25.252907 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.252913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.252936 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:03:25.252976 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.252982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.253148 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.253258 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.254206 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.255232 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.257127 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.257640 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.257735 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.261534 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.261546 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.261559 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.261586 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.264341 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.264973 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.264982 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.264985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.265101 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.265456 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.265463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.265466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.265821 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.266022 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.266510 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.266730 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":396,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab.
Entry for principal kudu/127.1.87.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab.
I20260619 13:03:25.269743 14960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:03:25.270534 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.124:34003
--webserver_interface=127.1.87.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab
--principal=kudu/127.1.87.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:39531,127.1.87.125:37003,127.1.87.124:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:03:25.270987 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.270487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40252)
0619 13:03:25.270714 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.270857 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 13:03:25.271366 14911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
I20260619 13:03:25.272948 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.272650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40262)
0619 13:03:25.272784 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.272856 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 13:03:25.273233 14960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:34003: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
I20260619 13:03:25.322566 14960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } attempt: 1
I20260619 13:03:25.323869 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.323279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40276)
0619 13:03:25.323553 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.323680 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 13:03:25.325011 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.324613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40292)
0619 13:03:25.324790 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.324877 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 13:03:25.325426 14960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:34003: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
I20260619 13:03:25.393746 14960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } attempt: 2
I20260619 13:03:25.395079 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.394446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40302)
0619 13:03:25.394714 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.394851 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 13:03:25.396389 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.395967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40318)
0619 13:03:25.396152 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.396244 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 13:03:25.396644 14960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:34003: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
I20260619 13:03:25.418008 14886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } attempt: 5
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
I20260619 13:03:25.433004 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.418898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50584)
0619 13:03:25.419146 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.419168 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:25.419296 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:25.419722 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:25.419727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:25.420257 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:25.420546 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.420561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.421552 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.421559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.422972 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.422987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.423851 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.423857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.424034 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.424701 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:25.424732 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:25.427503 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:25.430523 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.430528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.430533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.430917 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.431306 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.431332 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.431337 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.431481 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.432020 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:25.432025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:25.432172 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 13:03:25.432342 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 13:03:25.432648 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 13:03:25.432996 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.419069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.1.87.125:37003)
0619 13:03:25.419231 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.419235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.419255 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:03:25.419468 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.419472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.419558 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.419659 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.420718 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.421402 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.423130 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.423703 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.423803 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.427707 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.427712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.427715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.427747 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.430365 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.431045 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.431057 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.431060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.431182 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.431612 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.431618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.431621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.431899 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.432056 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.432452 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.432674 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 13:03:25.434087 14886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:03:25.435212 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.434821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40334)
0619 13:03:25.434995 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.435087 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 13:03:25.436185 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.435857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40348)
0619 13:03:25.436004 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.436091 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 13:03:25.436460 14886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:34003: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
I20260619 13:03:25.487720 14886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } attempt: 1
I20260619 13:03:25.487895 14960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } attempt: 3
I20260619 13:03:25.489009 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.488470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40362)
0619 13:03:25.488753 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.488851 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260619 13:03:25.489245 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.488470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40378)
0619 13:03:25.488875 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.489023 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 13:03:25.489997 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.489679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40394)
0619 13:03:25.489829 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.489903 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 13:03:25.490222 14886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:34003: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
I20260619 13:03:25.490507 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.489974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40402)
0619 13:03:25.490253 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.490350 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
W20260619 13:03:25.490746 14960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:34003: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
I20260619 13:03:25.558493 14886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } attempt: 2
I20260619 13:03:25.559720 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.559186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40416)
0619 13:03:25.559455 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.559546 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 13:03:25.560655 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.560335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40424)
0619 13:03:25.560474 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.560548 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:03:25.560858 14886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:34003: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
W20260619 13:03:25.605051 14967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:25.605396 14967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:25.605558 14967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:25.612848 14967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:25.612985 14967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:25.613095 14967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:25.613202 14967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:03:25.613288 14967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:25.623090 14967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:39531,127.1.87.125:37003,127.1.87.124:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab
--principal=kudu/127.1.87.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.124:34003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.87.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:25.624434 14967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:25.625722 14967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:03:25.633985 14960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } attempt: 4
I20260619 13:03:25.635150 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.634678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40436)
0619 13:03:25.634896 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.634985 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 13:03:25.636377 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.635935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40446)
0619 13:03:25.636103 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.636201 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 13:03:25.636698 14960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:34003: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
W20260619 13:03:25.637385 14973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:25.637641 14972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:25.637440 14975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:25.638821 14967 server_base.cc:1034] running on GCE node
I20260619 13:03:25.639631 14967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:25.640944 14967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:25.642141 14967 hybrid_clock.cc:648] HybridClock initialized: now 1781874205642102 us; error 38 us; skew 500 ppm
I20260619 13:03:25.642472 14967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874205, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:25.645421 14967 init.cc:377] Logged in from keytab as kudu/127.1.87.124@KRBTEST.COM (short username kudu)
I20260619 13:03:25.646898 14967 webserver.cc:460] Webserver started at http://127.1.87.124:37441/ using document root <none> and password file <none>
I20260619 13:03:25.647630 14967 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:25.647771 14967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:25.648056 14967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:25.650890 14967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/data/instance:
uuid: "56037c72c9474bbcb98151383944d9e0"
format_stamp: "Formatted at 2026-06-19 13:03:25 on dist-test-slave-tpsm"
server_key: "d1526b25c385c29582bf15105da32caa"
server_key_iv: "e7b9ece589ea680677ef03409ae3b409"
server_key_version: "encryptionkey@0"
I20260619 13:03:25.651896 14967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/wal/instance:
uuid: "56037c72c9474bbcb98151383944d9e0"
format_stamp: "Formatted at 2026-06-19 13:03:25 on dist-test-slave-tpsm"
server_key: "d1526b25c385c29582bf15105da32caa"
server_key_iv: "e7b9ece589ea680677ef03409ae3b409"
server_key_version: "encryptionkey@0"
I20260619 13:03:25.656688 14967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:03:25.659971 14982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:25.661216 14967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:25.661396 14967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/wal
uuid: "56037c72c9474bbcb98151383944d9e0"
format_stamp: "Formatted at 2026-06-19 13:03:25 on dist-test-slave-tpsm"
server_key: "d1526b25c385c29582bf15105da32caa"
server_key_iv: "e7b9ece589ea680677ef03409ae3b409"
server_key_version: "encryptionkey@0"
I20260619 13:03:25.661588 14967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:25.667028 14886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } attempt: 3
I20260619 13:03:25.668027 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.667615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40458)
0619 13:03:25.667798 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.667886 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 13:03:25.668968 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.668578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40462)
0619 13:03:25.668807 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.668877 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260619 13:03:25.669173 14886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:34003: Network error: Client connection negotiation failed: client connection to 127.1.87.124:34003: connect: Connection refused (error 111)
I20260619 13:03:25.680039 14967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:25.683921 14967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:25.684211 14967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:25.702479 14967 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.124:34003
I20260619 13:03:25.702524 15033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.124:34003 every 8 connection(s)
I20260619 13:03:25.703796 14967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
I20260619 13:03:25.707284 15034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:25.712718 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 14967
I20260619 13:03:25.712937 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874004469744-1373-0/minicluster-data/master-2/wal/instance
I20260619 13:03:25.713384 1373 external_mini_cluster.cc:1428] Setting key fb78410fe9afe8bfa8953f3a77890680
I20260619 13:03:25.715737 15034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874205, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
I20260619 13:03:25.735751 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.718068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.1.87.126:39531)
0619 13:03:25.718261 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.718266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.719091 (+ 825us) server_negotiation.cc:407] Connection header received
0619 13:03:25.719425 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.719431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.719513 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.719594 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.721401 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.722098 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.724250 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.724893 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.725026 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.729079 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.729084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.729087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.729129 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.732474 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.733211 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.733218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.733223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.733386 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.733863 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.733867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.733869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.734115 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.734309 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.735095 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.735376 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:03:25.736680 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.715029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.1.87.124:34003)
0619 13:03:25.715858 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.715872 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.715930 (+ 58us) server_negotiation.cc:407] Connection header received
0619 13:03:25.718532 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.718558 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.719044 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.719387 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.721223 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.722333 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.724199 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.725002 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.725160 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.730004 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.730014 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.730030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.730072 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.733513 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.734231 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.734239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.734242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.734397 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.734781 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.734784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.734786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.735207 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.735511 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.735972 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.736220 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":596,"thread_start_us":361,"threads_started":1}
I20260619 13:03:25.736716 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.718134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39531 (local address 127.0.0.1:58850)
0619 13:03:25.718951 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.718995 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:25.719218 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:25.719867 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:25.719877 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:25.720700 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:25.721240 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.721256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.722265 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.722271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.724109 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.724124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.725148 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.725155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.725373 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.726115 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:25.726144 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:25.728898 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:25.732601 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.732618 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.732627 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.733073 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.733497 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.733500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.733502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.733743 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.734275 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:25.734281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:25.734678 (+ 397us) client_negotiation.cc:769] Sending connection context
0619 13:03:25.734941 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 13:03:25.735375 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":3,"thread_start_us":448,"threads_started":1}
I20260619 13:03:25.738320 15034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874205, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:25.751008 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.738964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:37003 (local address 127.0.0.1:50590)
0619 13:03:25.739133 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.739154 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:25.739253 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:25.739749 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:25.739752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:25.740220 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:25.740373 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.740381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.741347 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.741354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.742214 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.742224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.742896 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.742901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.743098 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.743739 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:25.743755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:25.745591 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:25.748686 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.748691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.748696 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.749057 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.749446 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.749452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.749456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.749596 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.750072 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:25.750077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:25.750224 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 13:03:25.750360 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 13:03:25.750641 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 13:03:25.751034 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.739071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.1.87.125:37003)
0619 13:03:25.739259 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.739265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.739285 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:03:25.739378 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.739384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.739460 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.739542 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.740517 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.741211 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.742357 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.742757 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.742888 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.745778 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.745786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.745795 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.745827 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.748542 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.749178 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.749185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.749188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.749333 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.749708 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.749714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.749718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.749955 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.750107 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.750457 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.750661 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:03:25.752167 15034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:03:25.754256 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.740142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.1.87.126:39531)
0619 13:03:25.740338 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.740344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.740366 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:03:25.740403 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.740406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.740456 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.740555 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.741685 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.742405 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.743721 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.744113 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.744246 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.748489 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.748495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.748500 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.748539 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.751173 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.752013 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.752018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.752021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.752174 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.752549 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.752552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.752554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.752942 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.753376 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.753516 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.753736 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874205, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
I20260619 13:03:25.766507 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.753168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40478)
0619 13:03:25.753411 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.753428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:25.753528 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:25.753850 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:25.753856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:25.754327 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:25.754498 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.754507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.755631 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.755634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.756665 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.756677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.757652 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.757657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.757832 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.758468 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:25.758486 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:25.760519 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:25.763730 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.763734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.763736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.764121 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.764572 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.764575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.764577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.764702 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.765402 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:25.765406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:25.765577 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 13:03:25.765760 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 13:03:25.766085 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260619 13:03:25.766775 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.753163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.1.87.124:34003)
0619 13:03:25.753317 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.753323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.753505 (+ 182us) server_negotiation.cc:407] Connection header received
0619 13:03:25.753626 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.753631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.753704 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.753801 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.754627 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.755507 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.756774 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.757482 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.757620 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.760718 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.760723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.760727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.760755 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.763600 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.764242 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.764248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.764252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.764440 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.764818 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.764824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.764828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.765231 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.765342 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.765853 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.766304 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":41}
I20260619 13:03:25.771857 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.756096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.1.87.125:37003)
0619 13:03:25.756266 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.756272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.756287 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:03:25.756437 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.756442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.756506 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.756601 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.758130 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.758867 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.760183 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.760605 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.760740 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.765490 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.765502 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.765507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.765546 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.769289 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.770052 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.770055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.770057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.770187 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.770570 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.770573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.770575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.770819 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.771020 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.771426 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.771589 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:03:25.772254 15034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0: Bootstrap starting.
I20260619 13:03:25.776021 15034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:25.776672 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:03:25.777128 15034 log.cc:824] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:25.779877 15034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0: No bootstrap required, opened a new log
I20260619 13:03:25.789398 15034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } }
I20260619 13:03:25.789907 15034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:25.790017 15034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56037c72c9474bbcb98151383944d9e0, State: Initialized, Role: FOLLOWER
I20260619 13:03:25.790913 15034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } }
I20260619 13:03:25.795833 15047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:25.796221 15047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:03:25.797312 15034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:03:25.814301 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.785457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.1.87.124:34003)
0619 13:03:25.785693 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.785702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.794774 (+ 9072us) server_negotiation.cc:407] Connection header received
0619 13:03:25.794945 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.794952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.795040 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.795141 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.797632 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.798815 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.801275 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.802148 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.802302 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.805580 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.805587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.805591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.805643 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.810429 (+ 4786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.811425 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.811433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.811437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.811655 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.812140 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.812147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.812152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.812565 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.813494 (+ 929us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.813648 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.813884 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:03:25.816716 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.785028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.1.87.126:39531)
0619 13:03:25.785233 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.785241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.796032 (+ 10791us) server_negotiation.cc:407] Connection header received
0619 13:03:25.796329 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.796336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.796432 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.796512 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.798516 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.799568 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.805063 (+ 5495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.805765 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.805862 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.808945 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.808952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.808957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.809009 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.812560 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.813867 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.813873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.813876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.814027 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.814501 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.814508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.814513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.815043 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.815344 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.815917 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.816164 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 13:03:25.820130 15058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:03:25.820250 15058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:03:25.807554 14967 master_runner.cc:186] Error getting master registration for 127.1.87.126:39531: OK, instance_id { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" instance_seqno: 1781874204704797 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:03:25.821889 14967 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:03:25.822850 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.785364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.1.87.125:37003)
0619 13:03:25.785872 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.785880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.788198 (+ 2318us) server_negotiation.cc:407] Connection header received
0619 13:03:25.788280 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.788288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.788391 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.788501 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.791638 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.793139 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.798557 (+ 5418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.799207 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.799306 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.803847 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.803854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.803858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.803902 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.807194 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.820492 (+ 13298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.820499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.820502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.820712 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.821167 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.821173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.821176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.821578 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.821859 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.822228 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.822488 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362}
I20260619 13:03:25.846417 14886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } attempt: 4
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
I20260619 13:03:25.862901 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.847082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40498)
0619 13:03:25.847294 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.847353 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:25.847471 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:25.847901 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:25.847906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:25.848509 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:25.848732 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.848745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.850116 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.850122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.851600 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.851615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.852384 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.852390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.852571 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.853237 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:25.853260 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:25.855781 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:25.860417 (+ 4636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.860423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.860427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.860814 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.861222 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.861227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.861231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.861367 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.861896 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:25.861901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:25.862066 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 13:03:25.862200 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 13:03:25.862507 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 13:03:25.862888 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.847142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.1.87.124:34003)
0619 13:03:25.847370 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.847375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.847402 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:03:25.847610 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.847614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.847718 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.848885 (+ 1167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.849034 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.849978 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.851752 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.852231 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.852336 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.855974 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.855978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.855981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.856011 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.860278 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.860949 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.860957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.860959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.861100 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.861480 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.861489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.861491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.861764 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.861918 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.862297 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.862512 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:03:25.867820 14886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2: Bootstrap starting.
I20260619 13:03:25.871030 14886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:25.872156 14886 log.cc:824] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:25.874945 14886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2: No bootstrap required, opened a new log
I20260619 13:03:25.882862 14886 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } }
I20260619 13:03:25.883160 14886 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:25.883314 14886 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1944b05a7d84831abe21acd8a844ff2, State: Initialized, Role: FOLLOWER
I20260619 13:03:25.883890 14886 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } }
I20260619 13:03:25.884791 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:25.885164 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:03:25.885941 14886 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:03:25.896780 15072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:03:25.896896 15072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:03:25.895355 14819 master_runner.cc:186] Error getting master registration for 127.1.87.125:37003: OK, instance_id { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" instance_seqno: 1781874205203917 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:03:25.903543 14819 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:03:25.935974 14960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } attempt: 5
W20260619 13:03:25.936218 14988 tablet.cc:2369] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:03:25.942919 14840 tablet.cc:2369] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 13:03:25 dist-test-slave-tpsm krb5kdc[14806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874205, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
I20260619 13:03:25.951385 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.936802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:34003 (local address 127.0.0.1:40510)
0619 13:03:25.937081 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:25.937099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:25.937250 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:25.937641 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:25.937646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:25.938075 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:25.938232 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.938240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.939894 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.939899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.941209 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:25.941223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.942361 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.942365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:25.942501 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.943215 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:25.943241 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:25.945513 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:25.948875 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.948880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.948884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.949277 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.949668 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:25.949673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:25.949677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:25.949813 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:25.950357 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:25.950362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:25.950517 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 13:03:25.950666 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 13:03:25.950969 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260619 13:03:25.951421 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:25.936859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.1.87.124:34003)
0619 13:03:25.937034 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:25.937041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:25.937186 (+ 145us) server_negotiation.cc:407] Connection header received
0619 13:03:25.937397 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:25.937406 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:25.937485 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:25.937593 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:25.938383 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.939483 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.941377 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:25.942051 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:25.942186 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:25.945723 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:25.945728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:25.945732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:25.945762 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:25.948699 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.949401 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.949408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.949412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.949543 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:25.949924 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:25.949930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:25.949933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:25.950237 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:25.950394 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:25.950777 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 13:03:25.951006 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:03:25.956792 14960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9: Bootstrap starting.
I20260619 13:03:25.960269 14960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:25.961302 14960 log.cc:824] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:25.963932 14960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9: No bootstrap required, opened a new log
I20260619 13:03:25.971915 14960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } }
I20260619 13:03:25.972314 14960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:25.972434 14960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcc23a96b824502845a7c05d60657f9, State: Initialized, Role: FOLLOWER
I20260619 13:03:25.973186 14960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } }
I20260619 13:03:25.980165 15075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:25.980515 15075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:03:25.981933 14960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:03:26.002638 14893 master_runner.cc:263] No leader master found from master 1fcc23a96b824502845a7c05d60657f9
I20260619 13:03:26.002831 14893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 13:03:26.009838 15086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:03:26.009994 15086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:03:26.107744 15061 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:03:26.108083 15061 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } }
I20260619 13:03:26.109798 15061 leader_election.cc:288] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fcc23a96b824502845a7c05d60657f9 (127.1.87.125:37003), 56037c72c9474bbcb98151383944d9e0 (127.1.87.124:34003)
I20260619 13:03:26.111024 15009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1944b05a7d84831abe21acd8a844ff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56037c72c9474bbcb98151383944d9e0" is_pre_election: true
I20260619 13:03:26.111682 15009 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1944b05a7d84831abe21acd8a844ff2 in term 0.
I20260619 13:03:26.112715 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1944b05a7d84831abe21acd8a844ff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcc23a96b824502845a7c05d60657f9" is_pre_election: true
I20260619 13:03:26.113296 14935 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1944b05a7d84831abe21acd8a844ff2 in term 0.
I20260619 13:03:26.114076 14837 leader_election.cc:302] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56037c72c9474bbcb98151383944d9e0, f1944b05a7d84831abe21acd8a844ff2; no voters:
I20260619 13:03:26.115062 15061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:03:26.115206 15061 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:03:26.115301 15061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:26.118882 15061 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } }
I20260619 13:03:26.124526 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1944b05a7d84831abe21acd8a844ff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcc23a96b824502845a7c05d60657f9"
I20260619 13:03:26.124785 14935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:26.128463 14935 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1944b05a7d84831abe21acd8a844ff2 in term 1.
I20260619 13:03:26.129056 14835 leader_election.cc:302] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcc23a96b824502845a7c05d60657f9, f1944b05a7d84831abe21acd8a844ff2; no voters:
I20260619 13:03:26.132855 15009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1944b05a7d84831abe21acd8a844ff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56037c72c9474bbcb98151383944d9e0"
I20260619 13:03:26.133106 15009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:26.136822 15009 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1944b05a7d84831abe21acd8a844ff2 in term 1.
I20260619 13:03:26.137657 15061 leader_election.cc:288] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [CANDIDATE]: Term 1 election: Requested vote from peers 1fcc23a96b824502845a7c05d60657f9 (127.1.87.125:37003), 56037c72c9474bbcb98151383944d9e0 (127.1.87.124:34003)
I20260619 13:03:26.138083 15061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:03:26.138489 15061 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [term 1 LEADER]: Becoming Leader. State: Replica: f1944b05a7d84831abe21acd8a844ff2, State: Running, Role: LEADER
I20260619 13:03:26.139325 15061 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } }
I20260619 13:03:26.162078 15088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1944b05a7d84831abe21acd8a844ff2. Latest consensus state: current_term: 1 leader_uuid: "f1944b05a7d84831abe21acd8a844ff2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:26.162740 15088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:26.175967 15090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:03:26.178275 15090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 13:03:26.192387 14914 tablet.cc:2369] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:03:26.193879 15009 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [term 1 FOLLOWER]: Refusing update from remote peer f1944b05a7d84831abe21acd8a844ff2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:26.194972 15088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:03:26.202924 14935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [term 1 FOLLOWER]: Refusing update from remote peer f1944b05a7d84831abe21acd8a844ff2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:26.203975 15088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:03:26.226629 15047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1944b05a7d84831abe21acd8a844ff2. Latest consensus state: current_term: 1 leader_uuid: "f1944b05a7d84831abe21acd8a844ff2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:26.226943 15047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:03:26.234248 15091 mvcc.cc:205] Tried to move back new op lower bound from 7298556748558757888 to 7298556748359294976. Current Snapshot: MvccSnapshot[applied={T|T < 7298556748558757888}]
I20260619 13:03:26.242890 15047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1944b05a7d84831abe21acd8a844ff2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:26.243194 15047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:03:26.247852 15088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1944b05a7d84831abe21acd8a844ff2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:26.248124 15088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:26.251773 15090 catalog_manager.cc:1347] Generated new cluster ID: b76a9a632e9643c4912599a3e96ccfc8
I20260619 13:03:26.251878 15090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:03:26.275694 15075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1944b05a7d84831abe21acd8a844ff2. Latest consensus state: current_term: 1 leader_uuid: "f1944b05a7d84831abe21acd8a844ff2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:26.276089 15075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:03:26.281159 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1944b05a7d84831abe21acd8a844ff2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:26.281589 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:26.283699 15093 mvcc.cc:205] Tried to move back new op lower bound from 7298556748558757888 to 7298556748359294976. Current Snapshot: MvccSnapshot[applied={T|T < 7298556748558757888}]
I20260619 13:03:26.298573 15075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1944b05a7d84831abe21acd8a844ff2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1944b05a7d84831abe21acd8a844ff2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39531 } } peers { permanent_uuid: "1fcc23a96b824502845a7c05d60657f9" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 37003 } } peers { permanent_uuid: "56037c72c9474bbcb98151383944d9e0" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 34003 } } }
I20260619 13:03:26.298877 15075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:03:26.299708 15090 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:03:26.314842 15090 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:03:26.337751 15090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1944b05a7d84831abe21acd8a844ff2: Generated new TSK 0
I20260619 13:03:26.340456 15090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:03:26.822777 15058 catalog_manager.cc:1259] Loaded cluster ID: b76a9a632e9643c4912599a3e96ccfc8
I20260619 13:03:26.822891 15058 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0: loading cluster ID for follower catalog manager: success
I20260619 13:03:26.825623 15058 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0: acquiring CA information for follower catalog manager: success
I20260619 13:03:26.827513 15058 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56037c72c9474bbcb98151383944d9e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:03:27.012112 15086 catalog_manager.cc:1259] Loaded cluster ID: b76a9a632e9643c4912599a3e96ccfc8
I20260619 13:03:27.012218 15086 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9: loading cluster ID for follower catalog manager: success
I20260619 13:03:27.014607 15086 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9: acquiring CA information for follower catalog manager: success
I20260619 13:03:27.016373 15086 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1fcc23a96b824502845a7c05d60657f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:03:27.478777 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 14819
I20260619 13:03:27.480427 14881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:27.598688 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 14819
I20260619 13:03:27.620920 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 14893
I20260619 13:03:27.622114 14955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:27.739476 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 14893
I20260619 13:03:27.767654 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 14967
I20260619 13:03:27.768824 15029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:27.876756 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 14967
2026-06-19T13:03:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4958 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:03:27 dist-test-slave-tpsm krb5kdc[15110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:03:27 dist-test-slave-tpsm krb5kdc[15110](info): set up 2 sockets
Jun 19 13:03:27 dist-test-slave-tpsm krb5kdc[15110](info): commencing operation
krb5kdc: starting...
W20260619 13:03:28.996678 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:03:29 dist-test-slave-tpsm krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:03:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:03:29.233129 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37413
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:37413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:29.559204 15123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:29.559602 15123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:29.559770 15123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:29.567055 15123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:29.567164 15123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:29.567261 15123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:29.567394 15123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:03:29.567487 15123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:29.577399 15123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:37413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:29.578704 15123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:29.580041 15123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:29.592322 15128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:29.592305 15129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:29.592420 15131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:29.593566 15123 server_base.cc:1034] running on GCE node
I20260619 13:03:29.594419 15123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:29.595965 15123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:29.597226 15123 hybrid_clock.cc:648] HybridClock initialized: now 1781874209597181 us; error 52 us; skew 500 ppm
I20260619 13:03:29.597599 15123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:29 dist-test-slave-tpsm krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874209, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:29.600749 15123 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:03:29.602348 15123 webserver.cc:460] Webserver started at http://127.1.87.126:46431/ using document root <none> and password file <none>
I20260619 13:03:29.603062 15123 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:29.603210 15123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:29.603591 15123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:29.606758 15123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "81a00af02b0d4de9af4982f3f62253c1"
format_stamp: "Formatted at 2026-06-19 13:03:29 on dist-test-slave-tpsm"
server_key: "ea9f2162e22e7851fdf4226cf9628f9c"
server_key_iv: "65198e93c4d763c81b61845fbf183847"
server_key_version: "encryptionkey@0"
I20260619 13:03:29.607633 15123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "81a00af02b0d4de9af4982f3f62253c1"
format_stamp: "Formatted at 2026-06-19 13:03:29 on dist-test-slave-tpsm"
server_key: "ea9f2162e22e7851fdf4226cf9628f9c"
server_key_iv: "65198e93c4d763c81b61845fbf183847"
server_key_version: "encryptionkey@0"
I20260619 13:03:29.612790 15123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 13:03:29.616207 15138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:29.617444 15123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:03:29.617626 15123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "81a00af02b0d4de9af4982f3f62253c1"
format_stamp: "Formatted at 2026-06-19 13:03:29 on dist-test-slave-tpsm"
server_key: "ea9f2162e22e7851fdf4226cf9628f9c"
server_key_iv: "65198e93c4d763c81b61845fbf183847"
server_key_version: "encryptionkey@0"
I20260619 13:03:29.617830 15123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:29.629067 15123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:29.633188 15123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:29.633478 15123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:29.652916 15123 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:37413
I20260619 13:03:29.652945 15189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:37413 every 8 connection(s)
I20260619 13:03:29.654353 15123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:03:29.656497 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 15123
I20260619 13:03:29.656729 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:03:29.657536 1373 external_mini_cluster.cc:1428] Setting key c0b50b48c804527bd7de0846d348a5b6
I20260619 13:03:29.658756 15190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:29.672217 15190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1: Bootstrap starting.
I20260619 13:03:29.676225 15190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1: Neither blocks nor log segments found. Creating new log.
Jun 19 13:03:29 dist-test-slave-tpsm krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:29.677253 15190 log.cc:824] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:29.681284 15190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1: No bootstrap required, opened a new log
I20260619 13:03:29.687757 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:29.659470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.1.87.126:37413)
0619 13:03:29.660245 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:29.660256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:29.660296 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:03:29.661862 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:29.661883 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:29.662261 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:29.662598 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:29.664483 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:29.665875 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:29.667779 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:29.668491 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:29.668632 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:29.677680 (+ 9048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:29.677692 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:29.677714 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:29.677774 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:29.682034 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:29.682964 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:29.682973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:29.682984 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:29.683198 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:29.683841 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:29.683847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:29.683851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:29.684596 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:29.684911 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:29.686281 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0619 13:03:29.686651 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":365,"threads_started":1}
I20260619 13:03:29.690706 15190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a00af02b0d4de9af4982f3f62253c1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37413 } }
I20260619 13:03:29.691203 15190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:29.691382 15190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a00af02b0d4de9af4982f3f62253c1, State: Initialized, Role: FOLLOWER
I20260619 13:03:29.691818 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:03:29.692087 15190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a00af02b0d4de9af4982f3f62253c1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37413 } }
I20260619 13:03:29.692339 15190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:03:29.692479 15190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:03:29.692662 15190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:29.695076 15190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a00af02b0d4de9af4982f3f62253c1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37413 } }
I20260619 13:03:29.695634 15190 leader_election.cc:302] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81a00af02b0d4de9af4982f3f62253c1; no voters:
I20260619 13:03:29.696223 15190 leader_election.cc:288] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:03:29.696350 15195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:03:29.697052 15195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [term 1 LEADER]: Becoming Leader. State: Replica: 81a00af02b0d4de9af4982f3f62253c1, State: Running, Role: LEADER
I20260619 13:03:29.697525 15195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a00af02b0d4de9af4982f3f62253c1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37413 } }
I20260619 13:03:29.698426 15190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:03:29.700881 15197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81a00af02b0d4de9af4982f3f62253c1. Latest consensus state: current_term: 1 leader_uuid: "81a00af02b0d4de9af4982f3f62253c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a00af02b0d4de9af4982f3f62253c1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37413 } } }
I20260619 13:03:29.701498 15197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:29.702713 15196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81a00af02b0d4de9af4982f3f62253c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a00af02b0d4de9af4982f3f62253c1" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37413 } } }
I20260619 13:03:29.703364 15196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:29.705538 15202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:03:29.712458 15202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:03:29.722448 15202 catalog_manager.cc:1347] Generated new cluster ID: dc58f9e9dce34034978d86adc4f12da0
I20260619 13:03:29.722589 15202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:03:29.733299 15202 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:03:29.734656 15202 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:03:29.754256 15202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81a00af02b0d4de9af4982f3f62253c1: Generated new TSK 0
I20260619 13:03:29.755157 15202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:03:29.805227 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 15123
I20260619 13:03:29.806563 15185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:29.903872 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 15123
2026-06-19T13:03:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2022 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:03:29 dist-test-slave-tpsm krb5kdc[15218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:03:29 dist-test-slave-tpsm krb5kdc[15218](info): set up 2 sockets
Jun 19 13:03:29 dist-test-slave-tpsm krb5kdc[15218](info): commencing operation
krb5kdc: starting...
W20260619 13:03:31.007828 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:03:31 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:03:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:03:31.240871 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37767
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:37767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:31.566166 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:31.566469 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:31.566637 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:31.573984 15231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:31.574101 15231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:31.574204 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:31.574294 15231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:03:31.574364 15231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:31.584441 15231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:37767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:31.585870 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:31.587181 15231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:31.599097 15236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:31.599062 15237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:31.599146 15239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:31.599956 15231 server_base.cc:1034] running on GCE node
I20260619 13:03:31.600776 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:31.602195 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:31.603437 15231 hybrid_clock.cc:648] HybridClock initialized: now 1781874211603416 us; error 93 us; skew 500 ppm
I20260619 13:03:31.603798 15231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:31 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874211, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:31.607164 15231 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:03:31.608865 15231 webserver.cc:460] Webserver started at http://127.1.87.126:33655/ using document root <none> and password file <none>
I20260619 13:03:31.609587 15231 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:31.609745 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:31.610054 15231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:31.613153 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "8b1dd7ecfcab41d59efe9d9ff552db13"
format_stamp: "Formatted at 2026-06-19 13:03:31 on dist-test-slave-tpsm"
server_key: "4c612900d14f7b168d4408c6d9e73f5c"
server_key_iv: "1871d2454bb6707b9bbf6da9904a7e16"
server_key_version: "encryptionkey@0"
I20260619 13:03:31.613932 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "8b1dd7ecfcab41d59efe9d9ff552db13"
format_stamp: "Formatted at 2026-06-19 13:03:31 on dist-test-slave-tpsm"
server_key: "4c612900d14f7b168d4408c6d9e73f5c"
server_key_iv: "1871d2454bb6707b9bbf6da9904a7e16"
server_key_version: "encryptionkey@0"
I20260619 13:03:31.619273 15231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 13:03:31.622870 15246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:31.624251 15231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:31.624433 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "8b1dd7ecfcab41d59efe9d9ff552db13"
format_stamp: "Formatted at 2026-06-19 13:03:31 on dist-test-slave-tpsm"
server_key: "4c612900d14f7b168d4408c6d9e73f5c"
server_key_iv: "1871d2454bb6707b9bbf6da9904a7e16"
server_key_version: "encryptionkey@0"
I20260619 13:03:31.624640 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:31.644932 15231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:31.648813 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:31.649107 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:31.668560 15231 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:37767
I20260619 13:03:31.668613 15297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:37767 every 8 connection(s)
I20260619 13:03:31.669958 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:03:31.673565 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 15231
I20260619 13:03:31.673805 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:03:31.674165 15298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:31.674743 1373 external_mini_cluster.cc:1428] Setting key 664b032afb65513ca76e22ecf3cd1576
I20260619 13:03:31.689013 15298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Bootstrap starting.
Jun 19 13:03:31 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:31.692687 15298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:31.694078 15298 log.cc:824] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:31.697088 15298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: No bootstrap required, opened a new log
I20260619 13:03:31.699195 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:31.677014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.1.87.126:37767)
0619 13:03:31.677807 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:31.677818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:31.677861 (+ 43us) server_negotiation.cc:407] Connection header received
0619 13:03:31.679439 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:31.679466 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:31.679951 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:31.680407 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:31.681725 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:31.683751 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:31.685570 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:31.686658 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:31.686848 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:31.691164 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:31.691175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:31.691192 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:31.691236 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:31.695132 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:31.695904 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:31.695917 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:31.695924 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:31.696080 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:31.696487 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:31.696493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:31.696496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:31.696999 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:31.697366 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:31.697798 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 13:03:31.698236 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":389,"threads_started":1}
I20260619 13:03:31.705668 15298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:31.706146 15298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:31.706257 15298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1dd7ecfcab41d59efe9d9ff552db13, State: Initialized, Role: FOLLOWER
I20260619 13:03:31.706972 15298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:31.707227 15298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:03:31.707393 15298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:03:31.707576 15298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:31.710325 15298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:31.710873 15298 leader_election.cc:302] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b1dd7ecfcab41d59efe9d9ff552db13; no voters:
I20260619 13:03:31.711616 15298 leader_election.cc:288] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:03:31.711719 15303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:03:31.712456 15303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 1 LEADER]: Becoming Leader. State: Replica: 8b1dd7ecfcab41d59efe9d9ff552db13, State: Running, Role: LEADER
I20260619 13:03:31.713068 15303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:31.713889 15298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:03:31.716455 15305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1dd7ecfcab41d59efe9d9ff552db13. Latest consensus state: current_term: 1 leader_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } } }
I20260619 13:03:31.716781 15305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:31.716981 15304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } } }
I20260619 13:03:31.717545 15304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:31.724439 15309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:03:31.729058 15309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:03:31.738276 15309 catalog_manager.cc:1347] Generated new cluster ID: 5f8c3dfdc96449af8fff6d0520d4328a
I20260619 13:03:31.738468 15309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:03:31.771349 15309 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:03:31.772864 15309 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:03:31.790876 15309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Generated new TSK 0
I20260619 13:03:31.791850 15309 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:03:31.864202 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37767
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:32.205844 15324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:32.206159 15324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:32.206329 15324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:32.214936 15324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:32.215109 15324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:32.215251 15324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:03:32.226648 15324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:32.228096 15324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:32.230436 15324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:32.245061 15332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:32.245107 15330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:32.245154 15329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:32.245889 15324 server_base.cc:1034] running on GCE node
I20260619 13:03:32.246527 15324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:32.247924 15324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:32.249231 15324 hybrid_clock.cc:648] HybridClock initialized: now 1781874212249153 us; error 64 us; skew 500 ppm
I20260619 13:03:32.249655 15324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:32 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874212, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:32.253037 15324 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:03:32.254629 15324 webserver.cc:460] Webserver started at http://127.1.87.65:41797/ using document root <none> and password file <none>
I20260619 13:03:32.255415 15324 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:32.255563 15324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:32.255883 15324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:32.259052 15324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "4e5eda9a7f074f8683585b601e01c573"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "b74989bd4148710dc873e448f6462d8f"
server_key_iv: "7b455d496d45aabde166567927c4e029"
server_key_version: "encryptionkey@0"
I20260619 13:03:32.259963 15324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "4e5eda9a7f074f8683585b601e01c573"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "b74989bd4148710dc873e448f6462d8f"
server_key_iv: "7b455d496d45aabde166567927c4e029"
server_key_version: "encryptionkey@0"
I20260619 13:03:32.265424 15324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:03:32.269064 15339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:32.270457 15324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:03:32.270648 15324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "4e5eda9a7f074f8683585b601e01c573"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "b74989bd4148710dc873e448f6462d8f"
server_key_iv: "7b455d496d45aabde166567927c4e029"
server_key_version: "encryptionkey@0"
I20260619 13:03:32.270865 15324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:32.294950 15324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:32.300060 15324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:32.300457 15324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:32.301460 15324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:32.302904 15324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:32.303009 15324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:32.303156 15324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:32.303236 15324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:32.341675 15324 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:46719
I20260619 13:03:32.341742 15451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:46719 every 8 connection(s)
I20260619 13:03:32.342993 15324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:03:32.351662 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 15324
I20260619 13:03:32.351917 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:03:32.352603 1373 external_mini_cluster.cc:1428] Setting key 9d63a3976b625b27e259ce62dc6c07a5
Jun 19 13:03:32 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874212, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:32.364260 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:32.345583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:54953 (local address 127.1.87.126:37767)
0619 13:03:32.346024 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:32.346029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:32.347099 (+ 1070us) server_negotiation.cc:407] Connection header received
0619 13:03:32.348964 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:32.348971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:32.349064 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:32.349195 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:32.350733 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:32.351539 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:32.353204 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:32.353651 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:32.353815 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:32.358072 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:32.358077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:32.358080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:32.358113 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:32.360823 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:32.361757 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:32.361761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:32.361764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:32.361889 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:32.362442 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:32.362447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:32.362449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:32.362746 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:32.362873 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:32.363805 (+ 932us) server_negotiation.cc:299] Negotiation successful
0619 13:03:32.363989 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":223,"threads_started":1}
I20260619 13:03:32.365767 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:32.346108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:54953)
0619 13:03:32.346845 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:32.346886 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:32.348703 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:32.349431 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:32.349440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:32.349990 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:32.350566 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:32.350583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:32.351696 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:32.351700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:32.353045 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:32.353057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:32.353897 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:32.353904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:32.354139 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:32.354966 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:32.354998 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:32.357830 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:32.360989 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:32.360999 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:32.361017 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:32.361618 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:32.362007 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:32.362012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:32.362017 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:32.362290 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:32.362894 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:32.362907 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:32.363403 (+ 496us) client_negotiation.cc:769] Sending connection context
0619 13:03:32.363719 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 13:03:32.364317 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":301,"threads_started":1}
I20260619 13:03:32.367475 15452 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37767
I20260619 13:03:32.367807 15452 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:32.368518 15452 heartbeater.cc:511] Master 127.1.87.126:37767 requested a full tablet report, sending...
I20260619 13:03:32.370777 15263 ts_manager.cc:194] Registered new tserver with Master: 4e5eda9a7f074f8683585b601e01c573 (127.1.87.65:46719)
I20260619 13:03:32.372339 15263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:54953
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:03:32.404714 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37767
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:32.739380 15458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:32.739727 15458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:32.739897 15458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:32.747851 15458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:32.748010 15458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:32.748204 15458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:03:32.758111 15458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:32.759523 15458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:32.760856 15458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:32.773950 15466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:32.773913 15464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:32.773993 15463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:32.780158 15458 server_base.cc:1034] running on GCE node
I20260619 13:03:32.780900 15458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:32.782163 15458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:32.783449 15458 hybrid_clock.cc:648] HybridClock initialized: now 1781874212783399 us; error 33 us; skew 500 ppm
I20260619 13:03:32.783792 15458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:32 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874212, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:32.787075 15458 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:03:32.788728 15458 webserver.cc:460] Webserver started at http://127.1.87.66:39855/ using document root <none> and password file <none>
I20260619 13:03:32.789484 15458 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:32.789631 15458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:32.789932 15458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:32.793370 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "9dcf9c273103000932f0ece8bfdc847f"
server_key_iv: "16cdfb0850703093d8849e0edad14c7e"
server_key_version: "encryptionkey@0"
I20260619 13:03:32.794219 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "9dcf9c273103000932f0ece8bfdc847f"
server_key_iv: "16cdfb0850703093d8849e0edad14c7e"
server_key_version: "encryptionkey@0"
I20260619 13:03:32.799510 15458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 13:03:32.802963 15473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:32.804507 15458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:32.804698 15458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "9dcf9c273103000932f0ece8bfdc847f"
server_key_iv: "16cdfb0850703093d8849e0edad14c7e"
server_key_version: "encryptionkey@0"
I20260619 13:03:32.804929 15458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:32.840186 15458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:32.844834 15458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:32.845144 15458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:32.846345 15458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:32.847935 15458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:32.848042 15458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:32.848210 15458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:32.848284 15458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:32.885970 15458 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:37845
I20260619 13:03:32.886014 15585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:37845 every 8 connection(s)
I20260619 13:03:32.887403 15458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:03:32.888556 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 15458
I20260619 13:03:32.888741 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:03:32.889333 1373 external_mini_cluster.cc:1428] Setting key b7e5b60d1b292a2318dac6c295f6ae55
Jun 19 13:03:32 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874212, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:32.913220 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:32.891030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:44617 (local address 127.1.87.126:37767)
0619 13:03:32.891751 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:32.891758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:32.893417 (+ 1659us) server_negotiation.cc:407] Connection header received
0619 13:03:32.896515 (+ 3098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:32.896520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:32.896614 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:32.896751 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:32.899098 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:32.900118 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:32.901986 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:32.902770 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:32.902925 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:32.906629 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:32.906638 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:32.906646 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:32.906697 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:32.909865 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:32.910695 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:32.910701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:32.910709 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:32.910838 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:32.911384 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:32.911389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:32.911397 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:32.911679 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:32.911836 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:32.912564 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 13:03:32.912796 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":434,"threads_started":1}
I20260619 13:03:32.914353 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:32.892084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:44617)
0619 13:03:32.893190 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:32.893248 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:32.896066 (+ 2818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:32.897072 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:32.897087 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:32.897965 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:32.898833 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:32.898869 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:32.900288 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:32.900296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:32.901836 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:32.901848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:32.903037 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:32.903042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:32.903255 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:32.904112 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:32.904141 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:32.906349 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:32.910043 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:32.910053 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:32.910072 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:32.910540 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:32.910954 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:32.910959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:32.910963 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:32.911243 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:32.911808 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:32.911817 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:32.912188 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 13:03:32.912466 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 13:03:32.913038 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":484,"threads_started":1}
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:03:32.915884 15586 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37767
I20260619 13:03:32.916253 15586 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:32.917071 15586 heartbeater.cc:511] Master 127.1.87.126:37767 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
I20260619 13:03:32.918864 15263 ts_manager.cc:194] Registered new tserver with Master: d4ce2875ab0b4f2abe705bfb9ef8fbe8 (127.1.87.66:37845)
I20260619 13:03:32.920215 15263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:44617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:03:32.946844 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37767
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:33.295607 15592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:33.295964 15592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:33.296142 15592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:33.303872 15592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:33.304029 15592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:33.304224 15592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:03:33.314643 15592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:33.316442 15592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:33.317992 15592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:33.332597 15597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:33.332636 15598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:33.332556 15600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:33.334072 15592 server_base.cc:1034] running on GCE node
I20260619 13:03:33.334735 15592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:33.335776 15592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:33.336987 15592 hybrid_clock.cc:648] HybridClock initialized: now 1781874213336914 us; error 62 us; skew 500 ppm
I20260619 13:03:33.337458 15592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:33 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874213, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:33.341198 15592 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:03:33.346869 15592 webserver.cc:460] Webserver started at http://127.1.87.67:37285/ using document root <none> and password file <none>
I20260619 13:03:33.347744 15592 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:33.347896 15592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:33.348202 15592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:33.351276 15592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "669f5a63cb0d40229cd5a59d6d6d31e9"
format_stamp: "Formatted at 2026-06-19 13:03:33 on dist-test-slave-tpsm"
server_key: "ff327b6aa9c62f5c2828df7077378b71"
server_key_iv: "f0a0ce0c482153e5e113b2446d431ab3"
server_key_version: "encryptionkey@0"
I20260619 13:03:33.352173 15592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "669f5a63cb0d40229cd5a59d6d6d31e9"
format_stamp: "Formatted at 2026-06-19 13:03:33 on dist-test-slave-tpsm"
server_key: "ff327b6aa9c62f5c2828df7077378b71"
server_key_iv: "f0a0ce0c482153e5e113b2446d431ab3"
server_key_version: "encryptionkey@0"
I20260619 13:03:33.357784 15592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:03:33.362823 15607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:33.364602 15592 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:03:33.364832 15592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "669f5a63cb0d40229cd5a59d6d6d31e9"
format_stamp: "Formatted at 2026-06-19 13:03:33 on dist-test-slave-tpsm"
server_key: "ff327b6aa9c62f5c2828df7077378b71"
server_key_iv: "f0a0ce0c482153e5e113b2446d431ab3"
server_key_version: "encryptionkey@0"
I20260619 13:03:33.365064 15592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:33.375733 15452 heartbeater.cc:503] Master 127.1.87.126:37767 was elected leader, sending a full tablet report...
I20260619 13:03:33.379522 15592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:33.384992 15592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:33.385356 15592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:33.386391 15592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:33.387923 15592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:33.388031 15592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:33.388185 15592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:33.388257 15592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:33.428366 15592 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:46307
I20260619 13:03:33.428445 15719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:46307 every 8 connection(s)
I20260619 13:03:33.429801 15592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:03:33.431265 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 15592
I20260619 13:03:33.431481 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:03:33.431950 1373 external_mini_cluster.cc:1428] Setting key d518514083ec05760202f55a5d1da15b
Jun 19 13:03:33 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874213, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:33.455029 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.432572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:39197 (local address 127.1.87.126:37767)
0619 13:03:33.433025 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:33.433030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:33.434637 (+ 1607us) server_negotiation.cc:407] Connection header received
0619 13:03:33.437009 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:33.437015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:33.437115 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:33.437264 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:33.439667 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.440559 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.443061 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.443631 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.443758 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.448859 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:33.448865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:33.448869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:33.448910 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:33.451673 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:33.452526 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:33.452533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:33.452542 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:33.452662 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:33.453126 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:33.453133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:33.453141 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:33.453441 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:33.453604 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:33.454373 (+ 769us) server_negotiation.cc:299] Negotiation successful
0619 13:03:33.454620 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":221,"threads_started":1}
I20260619 13:03:33.456075 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.433184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:39197)
0619 13:03:33.434453 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:33.434506 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:33.436634 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:33.437575 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:33.437584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:33.438468 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:33.439297 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.439411 (+ 114us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.440800 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.440806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.442820 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.442837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.443928 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.443934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.444210 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.445224 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:33.445255 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:33.448612 (+ 3357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:33.451848 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:33.451859 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:33.451881 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:33.452382 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:33.452787 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:33.452792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:33.452796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:33.452995 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:33.453575 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:33.453584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:33.453979 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 13:03:33.454272 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 13:03:33.454847 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":683,"threads_started":1}
I20260619 13:03:33.457624 15720 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37767
I20260619 13:03:33.458004 15720 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:33.459021 15720 heartbeater.cc:511] Master 127.1.87.126:37767 requested a full tablet report, sending...
I20260619 13:03:33.461174 15263 ts_manager.cc:194] Registered new tserver with Master: 669f5a63cb0d40229cd5a59d6d6d31e9 (127.1.87.67:46307)
I20260619 13:03:33.462158 15263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:39197
I20260619 13:03:33.463195 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:03:33.482354 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.468539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.1.87.126:37767)
0619 13:03:33.468781 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:33.468786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:33.469118 (+ 332us) server_negotiation.cc:407] Connection header received
0619 13:03:33.469279 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:33.469292 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:33.469412 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:33.469522 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:33.470956 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.471784 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.473014 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.473467 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.473565 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.476365 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:33.476371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:33.476374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:33.476410 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:33.479071 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:33.480216 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:33.480220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:33.480222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:33.480350 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:33.480858 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:33.480863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:33.480865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:33.481280 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:33.481600 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:33.481913 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 13:03:33.482113 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 13:03:33.486869 15262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:03:33.494324 15262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:03:33.532549 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.507255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:37845 (local address 127.0.0.1:36484)
0619 13:03:33.522490 (+ 15235us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:33.522535 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:33.522800 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:33.524092 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:33.524104 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:33.524151 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:33.524772 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.524788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.526921 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.526927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.529534 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.529546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.531438 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.531445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.531673 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.531722 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 13:03:33.531865 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 13:03:33.532060 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15007,"mutex_wait_us":200,"thread_start_us":13293,"threads_started":1}
I20260619 13:03:33.533056 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.507749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.1.87.67:46307)
0619 13:03:33.520199 (+ 12450us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:33.520214 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:33.522668 (+ 2454us) server_negotiation.cc:407] Connection header received
0619 13:03:33.523253 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:33.523261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:33.523721 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:33.523913 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:33.525165 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.526737 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.529794 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.531925 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.532068 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.532286 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:33.532559 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 13:03:33.532728 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12142,"thread_start_us":12497,"threads_started":1}
I20260619 13:03:33.533488 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.507880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.1.87.66:37845)
0619 13:03:33.521659 (+ 13779us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:33.521667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:33.523021 (+ 1354us) server_negotiation.cc:407] Connection header received
0619 13:03:33.523160 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:33.523172 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:33.523533 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:33.523749 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:33.524990 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.526737 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.529711 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.531232 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.531403 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.531623 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:33.532966 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0619 13:03:33.533158 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13617,"thread_start_us":13472,"threads_started":1}
I20260619 13:03:33.533088 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.507228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:46307 (local address 127.0.0.1:50732)
0619 13:03:33.522522 (+ 15294us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:33.522546 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:33.522800 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:33.524264 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:33.524268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:33.524292 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:33.524972 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.524986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.526955 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.526961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.529652 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.529666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.532150 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.532156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.532360 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.532383 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:03:33.532507 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 13:03:33.532640 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14897,"mutex_wait_us":181,"thread_start_us":14152,"threads_started":1}
I20260619 13:03:33.533178 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.521943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:46719 (local address 127.0.0.1:50846)
0619 13:03:33.524153 (+ 2210us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:33.524173 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:33.524276 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:33.525992 (+ 1716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:33.525997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:33.526033 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:33.526428 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.526438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.528638 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.528644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.530479 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.530490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.532329 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.532335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.532509 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.532552 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 13:03:33.532618 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 13:03:33.532765 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2093,"mutex_wait_us":1071,"thread_start_us":1440,"threads_started":1}
I20260619 13:03:33.534194 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.522307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.1.87.65:46719)
0619 13:03:33.525050 (+ 2743us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:33.525059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:33.525088 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:03:33.525194 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:33.525199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:33.525531 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:33.526657 (+ 1126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:33.526877 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.528481 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.530633 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.532097 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.533306 (+ 1209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.533521 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:33.533599 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 13:03:33.533766 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2582,"thread_start_us":2398,"threads_started":1}
I20260619 13:03:33.537977 15521 tablet_service.cc:1467] Processing CreateTablet for tablet 715c27cff9ac480bb598bb5f5a8411e1 (DEFAULT_TABLE table=test-table [id=5655d92b286046249a0652e1b8cf9f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:33.537977 15655 tablet_service.cc:1467] Processing CreateTablet for tablet 715c27cff9ac480bb598bb5f5a8411e1 (DEFAULT_TABLE table=test-table [id=5655d92b286046249a0652e1b8cf9f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:33.538228 15387 tablet_service.cc:1467] Processing CreateTablet for tablet 715c27cff9ac480bb598bb5f5a8411e1 (DEFAULT_TABLE table=test-table [id=5655d92b286046249a0652e1b8cf9f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:33.539536 15655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 715c27cff9ac480bb598bb5f5a8411e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:33.539536 15521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 715c27cff9ac480bb598bb5f5a8411e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:33.540270 15387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 715c27cff9ac480bb598bb5f5a8411e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:33.553649 15739 tablet_bootstrap.cc:492] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: Bootstrap starting.
I20260619 13:03:33.556985 15739 tablet_bootstrap.cc:654] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:33.558995 15739 log.cc:824] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:33.560743 15740 tablet_bootstrap.cc:492] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Bootstrap starting.
I20260619 13:03:33.564144 15741 tablet_bootstrap.cc:492] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: Bootstrap starting.
I20260619 13:03:33.567063 15740 tablet_bootstrap.cc:654] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:33.567529 15741 tablet_bootstrap.cc:654] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:33.568602 15740 log.cc:824] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:33.568706 15741 log.cc:824] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:33.570851 15739 tablet_bootstrap.cc:492] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: No bootstrap required, opened a new log
I20260619 13:03:33.571225 15739 ts_tablet_manager.cc:1397] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20260619 13:03:33.571668 15741 tablet_bootstrap.cc:492] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: No bootstrap required, opened a new log
I20260619 13:03:33.571931 15741 ts_tablet_manager.cc:1397] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 13:03:33.578361 15740 tablet_bootstrap.cc:492] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: No bootstrap required, opened a new log
I20260619 13:03:33.578748 15740 ts_tablet_manager.cc:1397] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20260619 13:03:33.581640 15741 raft_consensus.cc:348] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:33.582161 15741 raft_consensus.cc:374] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:33.582306 15741 raft_consensus.cc:729] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669f5a63cb0d40229cd5a59d6d6d31e9, State: Initialized, Role: FOLLOWER
I20260619 13:03:33.583114 15741 consensus_queue.cc:260] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:33.583127 15739 raft_consensus.cc:348] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:33.583704 15739 raft_consensus.cc:374] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:33.583889 15739 raft_consensus.cc:729] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5eda9a7f074f8683585b601e01c573, State: Initialized, Role: FOLLOWER
I20260619 13:03:33.584497 15720 heartbeater.cc:503] Master 127.1.87.126:37767 was elected leader, sending a full tablet report...
I20260619 13:03:33.584775 15739 consensus_queue.cc:260] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:33.585059 15741 ts_tablet_manager.cc:1428] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260619 13:03:33.587510 15739 ts_tablet_manager.cc:1428] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260619 13:03:33.589957 15740 raft_consensus.cc:348] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:33.590471 15740 raft_consensus.cc:374] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:33.590576 15740 raft_consensus.cc:729] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ce2875ab0b4f2abe705bfb9ef8fbe8, State: Initialized, Role: FOLLOWER
I20260619 13:03:33.591218 15740 consensus_queue.cc:260] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:33.592571 15586 heartbeater.cc:503] Master 127.1.87.126:37767 was elected leader, sending a full tablet report...
I20260619 13:03:33.593097 15740 ts_tablet_manager.cc:1428] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260619 13:03:33.595463 15453 tablet.cc:2369] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:03:33.639976 15587 tablet.cc:2369] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:03:33.681919 15721 tablet.cc:2369] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:03:33.787473 15746 raft_consensus.cc:482] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:03:33.787866 15746 raft_consensus.cc:504] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:33.789804 15746 leader_election.cc:288] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 669f5a63cb0d40229cd5a59d6d6d31e9 (127.1.87.67:46307), d4ce2875ab0b4f2abe705bfb9ef8fbe8 (127.1.87.66:37845)
I20260619 13:03:33.797223 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.790581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:37279 (local address 127.1.87.66:37845)
0619 13:03:33.790806 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:33.790812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:33.791213 (+ 401us) server_negotiation.cc:407] Connection header received
0619 13:03:33.791438 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:33.791447 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:33.791551 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:33.791653 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:33.792297 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.793265 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.795210 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.796097 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.796260 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.796468 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:33.796716 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 13:03:33.796870 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 13:03:33.797371 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.790233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:55103 (local address 127.1.87.67:46307)
0619 13:03:33.790450 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:33.790456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:33.791499 (+ 1043us) server_negotiation.cc:407] Connection header received
0619 13:03:33.791689 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:33.791694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:33.791772 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:33.791873 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:33.792511 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.793415 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.795268 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.796094 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.796226 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.796349 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:33.796812 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 13:03:33.796953 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:03:33.797441 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.790335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:46307 (local address 127.1.87.65:55103)
0619 13:03:33.791387 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:33.791405 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:33.791544 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:33.791910 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:33.791913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:33.791934 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:33.792341 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.792357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.793580 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.793585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.795075 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.795088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.796327 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.796334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.796614 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.796667 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 13:03:33.796773 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:03:33.796898 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":2,"thread_start_us":633,"threads_started":1}
I20260619 13:03:33.797484 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.790199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:37845 (local address 127.1.87.65:37279)
0619 13:03:33.791067 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:33.791087 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:33.791253 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:33.791680 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:33.791686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:33.791712 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:33.792136 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.792146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.793404 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.793409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.795028 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:33.795045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.796363 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.796373 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:33.796554 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.796578 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:03:33.796656 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 13:03:33.796790 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":63,"thread_start_us":433,"threads_started":1}
I20260619 13:03:33.798615 15541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715c27cff9ac480bb598bb5f5a8411e1" candidate_uuid: "4e5eda9a7f074f8683585b601e01c573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" is_pre_election: true
I20260619 13:03:33.798635 15675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715c27cff9ac480bb598bb5f5a8411e1" candidate_uuid: "4e5eda9a7f074f8683585b601e01c573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" is_pre_election: true
I20260619 13:03:33.799125 15541 raft_consensus.cc:2454] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e5eda9a7f074f8683585b601e01c573 in term 0.
I20260619 13:03:33.799129 15675 raft_consensus.cc:2454] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e5eda9a7f074f8683585b601e01c573 in term 0.
I20260619 13:03:33.799844 15343 leader_election.cc:302] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5eda9a7f074f8683585b601e01c573, 669f5a63cb0d40229cd5a59d6d6d31e9; no voters:
I20260619 13:03:33.800279 15746 raft_consensus.cc:2781] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:03:33.800436 15746 raft_consensus.cc:482] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:03:33.800539 15746 raft_consensus.cc:3037] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:33.802755 15746 raft_consensus.cc:504] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:33.803481 15746 leader_election.cc:288] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [CANDIDATE]: Term 1 election: Requested vote from peers 669f5a63cb0d40229cd5a59d6d6d31e9 (127.1.87.67:46307), d4ce2875ab0b4f2abe705bfb9ef8fbe8 (127.1.87.66:37845)
I20260619 13:03:33.803973 15541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715c27cff9ac480bb598bb5f5a8411e1" candidate_uuid: "4e5eda9a7f074f8683585b601e01c573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8"
I20260619 13:03:33.803973 15675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715c27cff9ac480bb598bb5f5a8411e1" candidate_uuid: "4e5eda9a7f074f8683585b601e01c573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9"
I20260619 13:03:33.804208 15541 raft_consensus.cc:3037] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:33.804193 15675 raft_consensus.cc:3037] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:33.807444 15675 raft_consensus.cc:2454] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e5eda9a7f074f8683585b601e01c573 in term 1.
I20260619 13:03:33.807442 15541 raft_consensus.cc:2454] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e5eda9a7f074f8683585b601e01c573 in term 1.
I20260619 13:03:33.807894 15343 leader_election.cc:302] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5eda9a7f074f8683585b601e01c573, 669f5a63cb0d40229cd5a59d6d6d31e9; no voters:
I20260619 13:03:33.808192 15746 raft_consensus.cc:2781] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:03:33.808696 15746 raft_consensus.cc:686] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 1 LEADER]: Becoming Leader. State: Replica: 4e5eda9a7f074f8683585b601e01c573, State: Running, Role: LEADER
I20260619 13:03:33.809209 15746 consensus_queue.cc:237] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:33.813660 15262 catalog_manager.cc:5495] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e5eda9a7f074f8683585b601e01c573 (127.1.87.65). New cstate: current_term: 1 leader_uuid: "4e5eda9a7f074f8683585b601e01c573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } health_report { overall_health: HEALTHY } } }
I20260619 13:03:33.836205 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.830292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.1.87.65:46719)
0619 13:03:33.830525 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:33.830533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:33.830636 (+ 103us) server_negotiation.cc:407] Connection header received
0619 13:03:33.830891 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:33.830895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:33.830973 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:33.831143 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:03:33.831652 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.832354 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.833908 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.834353 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.834500 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.835017 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:03:33.835609 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:03:33.835777 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:33.835834 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 13:03:33.835918 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:03:33.845045 15675 raft_consensus.cc:1261] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 1 FOLLOWER]: Refusing update from remote peer 4e5eda9a7f074f8683585b601e01c573: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:33.845353 15541 raft_consensus.cc:1261] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 1 FOLLOWER]: Refusing update from remote peer 4e5eda9a7f074f8683585b601e01c573: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:33.846160 15750 consensus_queue.cc:1035] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [LEADER]: Connected to new peer: Peer: permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:03:33.846567 15746 consensus_queue.cc:1035] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:03:33.858762 15753 mvcc.cc:205] Tried to move back new op lower bound from 7298556779897745408 to 7298556779766403072. Current Snapshot: MvccSnapshot[applied={T|T < 7298556779897745408}]
I20260619 13:03:33.859123 15754 mvcc.cc:205] Tried to move back new op lower bound from 7298556779897745408 to 7298556779766403072. Current Snapshot: MvccSnapshot[applied={T|T < 7298556779897745408}]
I20260619 13:03:33.879051 15262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:03:33.879397 15262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:03:33.883901 15262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Sending DeleteTablet for 3 replicas of tablet 715c27cff9ac480bb598bb5f5a8411e1
I20260619 13:03:33.885106 15655 tablet_service.cc:1514] Processing DeleteTablet for tablet 715c27cff9ac480bb598bb5f5a8411e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:33 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:50732
I20260619 13:03:33.885241 15521 tablet_service.cc:1514] Processing DeleteTablet for tablet 715c27cff9ac480bb598bb5f5a8411e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:33 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:36484
I20260619 13:03:33.885365 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 15324
I20260619 13:03:33.886097 15387 tablet_service.cc:1514] Processing DeleteTablet for tablet 715c27cff9ac480bb598bb5f5a8411e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:33 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:50846
I20260619 13:03:33.894621 15765 tablet_replica.cc:330] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: stopping tablet replica
I20260619 13:03:33.895939 15765 raft_consensus.cc:2229] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:03:33.897426 15765 raft_consensus.cc:2258] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:33.901829 15765 ts_tablet_manager.cc:1905] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:33.903113 15764 tablet_replica.cc:330] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: stopping tablet replica
I20260619 13:03:33.903889 15764 raft_consensus.cc:2229] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:03:33.904562 15763 tablet_replica.cc:330] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: stopping tablet replica
I20260619 13:03:33.908519 15764 raft_consensus.cc:2258] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:33.908552 15763 raft_consensus.cc:2229] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:03:33.909401 15763 raft_consensus.cc:2258] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:33.912346 15763 ts_tablet_manager.cc:1905] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:33.912703 15765 ts_tablet_manager.cc:1918] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:33.912941 15765 log.cc:1196] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/715c27cff9ac480bb598bb5f5a8411e1
I20260619 13:03:33.913641 15765 ts_tablet_manager.cc:1939] T 715c27cff9ac480bb598bb5f5a8411e1 P 4e5eda9a7f074f8683585b601e01c573: Deleting consensus metadata
I20260619 13:03:33.914048 15764 ts_tablet_manager.cc:1905] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:33.915742 15249 catalog_manager.cc:4841] TS 4e5eda9a7f074f8683585b601e01c573 (127.1.87.65:46719): tablet 715c27cff9ac480bb598bb5f5a8411e1 (table test-table [id=5655d92b286046249a0652e1b8cf9f24]) successfully deleted
Jun 19 13:03:33 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:03:33.921676 15764 ts_tablet_manager.cc:1918] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:33.922012 15764 log.cc:1196] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/715c27cff9ac480bb598bb5f5a8411e1
I20260619 13:03:33.922618 15764 ts_tablet_manager.cc:1939] T 715c27cff9ac480bb598bb5f5a8411e1 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Deleting consensus metadata
I20260619 13:03:33.924546 15249 catalog_manager.cc:4841] TS d4ce2875ab0b4f2abe705bfb9ef8fbe8 (127.1.87.66:37845): tablet 715c27cff9ac480bb598bb5f5a8411e1 (table test-table [id=5655d92b286046249a0652e1b8cf9f24]) successfully deleted
I20260619 13:03:33.925110 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:33.888725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.1.87.65:46719)
0619 13:03:33.888924 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:33.888931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:33.900261 (+ 11330us) server_negotiation.cc:407] Connection header received
0619 13:03:33.900906 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:33.900913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:33.901035 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:33.901131 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:33.904799 (+ 3668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.906210 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.910551 (+ 4341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:33.911389 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:33.911510 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:33.917183 (+ 5673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:33.917203 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:33.917220 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:33.917271 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:33.921463 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:33.922425 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:33.922434 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:33.922438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:33.922624 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:33.923086 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:33.923090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:33.923093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:33.923607 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:33.923892 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:33.924478 (+ 586us) server_negotiation.cc:299] Negotiation successful
0619 13:03:33.924733 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:03:33.925866 15763 ts_tablet_manager.cc:1918] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:33.926131 15763 log.cc:1196] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/715c27cff9ac480bb598bb5f5a8411e1
I20260619 13:03:33.926574 15447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:33.927884 15763 ts_tablet_manager.cc:1939] T 715c27cff9ac480bb598bb5f5a8411e1 P 669f5a63cb0d40229cd5a59d6d6d31e9: Deleting consensus metadata
I20260619 13:03:33.930083 15250 catalog_manager.cc:4841] TS 669f5a63cb0d40229cd5a59d6d6d31e9 (127.1.87.67:46307): tablet 715c27cff9ac480bb598bb5f5a8411e1 (table test-table [id=5655d92b286046249a0652e1b8cf9f24]) successfully deleted
I20260619 13:03:34.028553 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 15324
I20260619 13:03:34.046989 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 15458
Jun 19 13:03:34 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:03:34.063684 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:34.047829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.1.87.66:37845)
0619 13:03:34.047997 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:34.048002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:34.048019 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:03:34.048105 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:34.048109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:34.048178 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:34.048277 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:34.049550 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:34.050403 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:34.051981 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:34.052392 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:34.052494 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:34.056897 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:34.056912 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:34.056934 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:34.056970 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:34.059983 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:34.060847 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:34.060854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:34.060857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:34.060997 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:34.061516 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:34.061522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:34.061525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:34.062155 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:34.062621 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:34.062846 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 13:03:34.063091 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:03:34.064468 15581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:34.157917 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 15458
I20260619 13:03:34.175680 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 15592
Jun 19 13:03:34 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:03:34.190574 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:34.176455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.1.87.67:46307)
0619 13:03:34.176648 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:34.176656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:34.176676 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:03:34.176883 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:34.176895 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:34.176971 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:34.177078 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:34.178353 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:34.179036 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:34.180367 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:34.180748 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:34.180875 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:34.185371 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:34.185385 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:34.185402 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:34.185435 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:34.188274 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:34.188887 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:34.188893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:34.188896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:34.189038 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:34.189356 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:34.189359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:34.189361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:34.189701 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:34.189918 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:34.190155 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 13:03:34.190311 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:03:34.191231 15715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:34.282883 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 15592
I20260619 13:03:34.299724 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 15231
I20260619 13:03:34.301213 15293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:34.414242 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 15231
I20260619 13:03:34.436336 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37767
--webserver_interface=127.1.87.126
--webserver_port=33655
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:37767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:34.775970 15771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:34.776289 15771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:34.776451 15771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:34.784459 15771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:34.784593 15771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:34.784693 15771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:34.784811 15771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:03:34.784893 15771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:34.795308 15771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:37767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=33655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:34.796685 15771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:34.798007 15771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:34.821106 15778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:34.821167 15780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:34.821033 15777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:34.822297 15771 server_base.cc:1034] running on GCE node
I20260619 13:03:34.823233 15771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:34.824895 15771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:34.826157 15771 hybrid_clock.cc:648] HybridClock initialized: now 1781874214826122 us; error 61 us; skew 500 ppm
I20260619 13:03:34.826601 15771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:34 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874214, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:34.834059 15771 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:03:34.836021 15771 webserver.cc:460] Webserver started at http://127.1.87.126:33655/ using document root <none> and password file <none>
I20260619 13:03:34.836766 15771 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:34.836889 15771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:34.843017 15771 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:03:34.846016 15787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:34.847608 15771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:34.847787 15771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "8b1dd7ecfcab41d59efe9d9ff552db13"
format_stamp: "Formatted at 2026-06-19 13:03:31 on dist-test-slave-tpsm"
server_key: "4c612900d14f7b168d4408c6d9e73f5c"
server_key_iv: "1871d2454bb6707b9bbf6da9904a7e16"
server_key_version: "encryptionkey@0"
I20260619 13:03:34.848922 15771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:34.867638 15771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:34.872598 15771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:34.872915 15771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:34.894479 15771 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:37767
I20260619 13:03:34.894482 15838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:37767 every 8 connection(s)
I20260619 13:03:34.896404 15771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:03:34.903441 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 15771
I20260619 13:03:34.904443 15839 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:03:34.904444 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:46719
--local_ip_for_outbound_sockets=127.1.87.65
--tserver_master_addrs=127.1.87.126:37767
--webserver_port=41797
--webserver_interface=127.1.87.65
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:03:34.910032 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Bootstrap starting.
I20260619 13:03:34.931463 15839 log.cc:824] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:34.943918 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:03:34.944470 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Bootstrap complete.
I20260619 13:03:34.954038 15839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:34.954909 15839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1dd7ecfcab41d59efe9d9ff552db13, State: Initialized, Role: FOLLOWER
I20260619 13:03:34.955541 15839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:34.955793 15839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:03:34.955930 15839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:03:34.956100 15839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:03:34.959998 15839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:34.960539 15839 leader_election.cc:302] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b1dd7ecfcab41d59efe9d9ff552db13; no voters:
I20260619 13:03:34.961186 15839 leader_election.cc:288] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:03:34.961351 15843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:03:34.962826 15843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 2 LEADER]: Becoming Leader. State: Replica: 8b1dd7ecfcab41d59efe9d9ff552db13, State: Running, Role: LEADER
I20260619 13:03:34.963529 15843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:34.963717 15839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:03:34.966503 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1dd7ecfcab41d59efe9d9ff552db13. Latest consensus state: current_term: 2 leader_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } } }
I20260619 13:03:34.966904 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:34.967509 15844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } } }
I20260619 13:03:34.967795 15844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:34.978587 15849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:03:34.988408 15849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:03:34.989976 15849 catalog_manager.cc:1259] Loaded cluster ID: 5f8c3dfdc96449af8fff6d0520d4328a
I20260619 13:03:34.990144 15849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:03:34.993424 15849 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 13:03:34.995146 15860 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 13:03:35.011492 15849 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 13:03:35.022239 15849 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:03:35.024082 15849 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Loaded TSK: 0
I20260619 13:03:35.025605 15849 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:03:35.274147 15841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:35.274477 15841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:35.274631 15841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:35.282193 15841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:35.282356 15841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:35.282531 15841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:03:35.293005 15841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:46719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=41797
--tserver_master_addrs=127.1.87.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:35.294389 15841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:35.295822 15841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:35.309630 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:35.309660 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:35.309953 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:35.310374 15841 server_base.cc:1034] running on GCE node
I20260619 13:03:35.310999 15841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:35.312106 15841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:35.313329 15841 hybrid_clock.cc:648] HybridClock initialized: now 1781874215313282 us; error 41 us; skew 500 ppm
I20260619 13:03:35.313643 15841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:35 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874215, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:35.316836 15841 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:03:35.318403 15841 webserver.cc:460] Webserver started at http://127.1.87.65:41797/ using document root <none> and password file <none>
I20260619 13:03:35.319168 15841 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:35.319314 15841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:35.324810 15841 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 13:03:35.327569 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:35.329018 15841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:03:35.329209 15841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "4e5eda9a7f074f8683585b601e01c573"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "b74989bd4148710dc873e448f6462d8f"
server_key_iv: "7b455d496d45aabde166567927c4e029"
server_key_version: "encryptionkey@0"
I20260619 13:03:35.330166 15841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:35.348661 15841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:35.352917 15841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:35.353235 15841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:35.354633 15841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:35.356099 15841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:35.356215 15841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:35.356354 15841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:35.356454 15841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:35.390988 15841 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:46719
I20260619 13:03:35.391037 15988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:46719 every 8 connection(s)
I20260619 13:03:35.392423 15841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:03:35.400771 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 15841
I20260619 13:03:35.401407 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:37845
--local_ip_for_outbound_sockets=127.1.87.66
--tserver_master_addrs=127.1.87.126:37767
--webserver_port=39855
--webserver_interface=127.1.87.66
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:03:35 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874215, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:35.416481 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:35.395486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:54739)
0619 13:03:35.396628 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:35.396678 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:35.398447 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:35.399925 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:35.399938 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:35.400569 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:35.401149 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:35.401166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:35.402857 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:35.402869 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:35.404149 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:35.404160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:35.405184 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:35.405192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:35.405471 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:35.406179 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:35.406202 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:35.408413 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:35.412002 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:35.412012 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:35.412027 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:35.412449 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:35.412912 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:35.412916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:35.412920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:35.413096 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:35.413859 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:35.413867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:35.414205 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 13:03:35.414476 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 13:03:35.415049 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":383,"threads_started":1}
I20260619 13:03:35.416481 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:35.395220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:54739 (local address 127.1.87.126:37767)
0619 13:03:35.396252 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:35.396278 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:35.396818 (+ 540us) server_negotiation.cc:407] Connection header received
0619 13:03:35.398979 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:35.398991 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:35.399400 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:35.399772 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:35.401384 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:35.402687 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:35.404302 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:35.404942 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:35.405155 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:35.408685 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:35.408693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:35.408707 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:35.408744 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:35.411851 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:35.412603 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:35.412610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:35.412617 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:35.412787 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:35.413224 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:35.413231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:35.413239 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:35.413717 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:35.413924 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:35.414683 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 13:03:35.415049 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":413,"threads_started":1}
I20260619 13:03:35.420331 15989 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37767
I20260619 13:03:35.420696 15989 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:35.421494 15989 heartbeater.cc:511] Master 127.1.87.126:37767 requested a full tablet report, sending...
I20260619 13:03:35.424232 15804 ts_manager.cc:194] Registered new tserver with Master: 4e5eda9a7f074f8683585b601e01c573 (127.1.87.65:46719)
I20260619 13:03:35.426386 15804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:54739
W20260619 13:03:35.738500 15993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:35.738781 15993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:35.738876 15993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:35.746194 15993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:35.746311 15993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:35.746426 15993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:03:35.756618 15993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:37845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=39855
--tserver_master_addrs=127.1.87.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:35.757893 15993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:35.759135 15993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:35.772840 15998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:35.773031 16001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:35.773011 15999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:35.774036 15993 server_base.cc:1034] running on GCE node
I20260619 13:03:35.774747 15993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:35.775835 15993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:35.777136 15993 hybrid_clock.cc:648] HybridClock initialized: now 1781874215777072 us; error 64 us; skew 500 ppm
I20260619 13:03:35.777496 15993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:35 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874215, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:35.781176 15993 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:03:35.782804 15993 webserver.cc:460] Webserver started at http://127.1.87.66:39855/ using document root <none> and password file <none>
I20260619 13:03:35.783566 15993 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:35.783694 15993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:35.789469 15993 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:03:35.792384 16008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:35.793870 15993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:35.794013 15993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "9dcf9c273103000932f0ece8bfdc847f"
server_key_iv: "16cdfb0850703093d8849e0edad14c7e"
server_key_version: "encryptionkey@0"
I20260619 13:03:35.795140 15993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:35.809407 15993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:35.813861 15993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:35.814117 15993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:35.815181 15993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:35.816712 15993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:35.816802 15993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:35.816936 15993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:35.817008 15993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:35.852754 15993 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:37845
I20260619 13:03:35.852800 16120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:37845 every 8 connection(s)
I20260619 13:03:35.854277 15993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:03:35.855789 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 15993
I20260619 13:03:35.856542 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:46307
--local_ip_for_outbound_sockets=127.1.87.67
--tserver_master_addrs=127.1.87.126:37767
--webserver_port=37285
--webserver_interface=127.1.87.67
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:03:35 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874215, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:35.880353 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:35.857483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:49423 (local address 127.1.87.126:37767)
0619 13:03:35.857712 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:35.857718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:35.858919 (+ 1201us) server_negotiation.cc:407] Connection header received
0619 13:03:35.861918 (+ 2999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:35.861923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:35.861995 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:35.862111 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:35.864315 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:35.865632 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:35.868358 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:35.868797 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:35.868910 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:35.873615 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:35.873621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:35.873624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:35.873664 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:35.876304 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:35.877472 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:35.877478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:35.877481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:35.877665 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:35.878237 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:35.878240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:35.878243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:35.878617 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:35.878750 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:35.879827 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0619 13:03:35.880014 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:03:35.882006 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:35.857880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:49423)
0619 13:03:35.858740 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:35.858787 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:35.861571 (+ 2784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:35.862429 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:35.862443 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:35.863247 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:35.864113 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:35.864141 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:35.865838 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:35.865843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:35.868153 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:35.868171 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:35.869046 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:35.869056 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:35.869418 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:35.870195 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:35.870230 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:35.873312 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:35.876473 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:35.876487 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:35.876512 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:35.877259 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:35.877811 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:35.877819 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:35.877825 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:35.878097 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:35.878769 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:35.878781 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:35.879298 (+ 517us) client_negotiation.cc:769] Sending connection context
0619 13:03:35.879716 (+ 418us) client_negotiation.cc:240] Negotiation successful
0619 13:03:35.880352 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":331,"threads_started":1}
I20260619 13:03:35.883747 16121 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37767
I20260619 13:03:35.884120 16121 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:35.885046 16121 heartbeater.cc:511] Master 127.1.87.126:37767 requested a full tablet report, sending...
I20260619 13:03:35.886659 15804 ts_manager.cc:194] Registered new tserver with Master: d4ce2875ab0b4f2abe705bfb9ef8fbe8 (127.1.87.66:37845)
I20260619 13:03:35.887565 15804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:49423
W20260619 13:03:36.188442 16124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:36.188794 16124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:36.188972 16124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:36.196398 16124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:36.196565 16124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:36.196756 16124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:03:36.206986 16124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:46307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=37285
--tserver_master_addrs=127.1.87.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:36.208397 16124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:36.209735 16124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:36.223387 16129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:36.223405 16130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:36.223488 16132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:36.224921 16124 server_base.cc:1034] running on GCE node
I20260619 13:03:36.226006 16124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:36.227463 16124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:36.228792 16124 hybrid_clock.cc:648] HybridClock initialized: now 1781874216228731 us; error 41 us; skew 500 ppm
I20260619 13:03:36.229254 16124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:36 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874216, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:36.232736 16124 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:03:36.234650 16124 webserver.cc:460] Webserver started at http://127.1.87.67:37285/ using document root <none> and password file <none>
I20260619 13:03:36.235421 16124 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:36.235577 16124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:36.242323 16124 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:03:36.246882 16139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:36.248780 16124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:03:36.249007 16124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "669f5a63cb0d40229cd5a59d6d6d31e9"
format_stamp: "Formatted at 2026-06-19 13:03:33 on dist-test-slave-tpsm"
server_key: "ff327b6aa9c62f5c2828df7077378b71"
server_key_iv: "f0a0ce0c482153e5e113b2446d431ab3"
server_key_version: "encryptionkey@0"
I20260619 13:03:36.250072 16124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:36.272090 16124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:36.276814 16124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:36.277096 16124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:36.278102 16124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:36.279556 16124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:36.279690 16124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:36.279810 16124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:36.279901 16124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:36.313678 16124 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:46307
I20260619 13:03:36.313766 16251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:46307 every 8 connection(s)
I20260619 13:03:36.315102 16124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:03:36.324883 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 16124
Jun 19 13:03:36 dist-test-slave-tpsm krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874216, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:36.339134 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.317953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:45583 (local address 127.1.87.126:37767)
0619 13:03:36.318133 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.318138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.319730 (+ 1592us) server_negotiation.cc:407] Connection header received
0619 13:03:36.322290 (+ 2560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.322295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.322378 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.322481 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:36.324513 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.325588 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.327835 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.328265 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.328404 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.332094 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:36.332099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:36.332104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:36.332146 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:36.335597 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.336538 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.336542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.336544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.336668 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.337197 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.337200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.337202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.337487 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:36.337596 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.338527 (+ 931us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.338741 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:03:36.340134 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.318375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:45583)
0619 13:03:36.319563 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:36.319600 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:36.321895 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:36.322748 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:36.322758 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:36.323536 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:36.324277 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.324297 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.325795 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.325804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.327676 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.327695 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.328514 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.328521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.328761 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.329515 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:36.329544 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:36.331862 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:36.335731 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:36.335743 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:36.335763 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:36.336403 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:36.336792 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:36.336797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:36.336802 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:36.337069 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:36.337618 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:36.337628 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:36.338090 (+ 462us) client_negotiation.cc:769] Sending connection context
0619 13:03:36.338404 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 13:03:36.338810 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":268,"threads_started":1}
I20260619 13:03:36.341463 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.325923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.1.87.126:37767)
0619 13:03:36.327045 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.327064 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.327084 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:03:36.327140 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.327144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.327201 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.327317 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:36.328831 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.329786 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.331126 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.331659 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.331793 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.335169 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:36.335174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:36.335177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:36.335217 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:36.338253 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.339085 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.339090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.339093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.339250 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.339853 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.339858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.339861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.340204 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:36.340450 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.340885 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.341108 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":742,"threads_started":1}
I20260619 13:03:36.341696 16252 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37767
I20260619 13:03:36.342012 16252 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:36.342731 16252 heartbeater.cc:511] Master 127.1.87.126:37767 requested a full tablet report, sending...
I20260619 13:03:36.344321 15804 ts_manager.cc:194] Registered new tserver with Master: 669f5a63cb0d40229cd5a59d6d6d31e9 (127.1.87.67:46307)
I20260619 13:03:36.345059 15804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:45583
I20260619 13:03:36.356478 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:03:36.363858 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.357334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.1.87.126:37767)
0619 13:03:36.357556 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.357565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.357929 (+ 364us) server_negotiation.cc:407] Connection header received
0619 13:03:36.358120 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.358124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.358256 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.358714 (+ 458us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:03:36.359098 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.359980 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.361593 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.362045 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.362156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.362702 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:03:36.363217 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:03:36.363427 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.363470 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.363542 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:03:36.364977 15804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:03:36.372617 15804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:03:36.396889 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.384879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:37845 (local address 127.0.0.1:36498)
0619 13:03:36.386779 (+ 1900us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:36.386826 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:36.387014 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:36.388772 (+ 1758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:36.388779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:36.388810 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:36.389242 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.389254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.391540 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.391546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.393995 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.394009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.395938 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.395944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.396143 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.396204 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 13:03:36.396313 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 13:03:36.396510 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"thread_start_us":3099,"threads_started":1}
I20260619 13:03:36.398283 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.385077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.1.87.66:37845)
0619 13:03:36.388127 (+ 3050us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.388134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.388163 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:03:36.388258 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.388266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.388522 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.389485 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:36.389726 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.391371 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.394207 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.395714 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.397440 (+ 1726us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.397658 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.397750 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.397928 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2892,"thread_start_us":2791,"threads_started":1}
I20260619 13:03:36.399111 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.385379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:46307 (local address 127.0.0.1:50748)
0619 13:03:36.388270 (+ 2891us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:36.388291 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:36.388493 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:36.389242 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:36.389246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:36.389287 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:36.389648 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.389661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.392779 (+ 3118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.392785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.394938 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.394953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.398226 (+ 3273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.398232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.398437 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.398477 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 13:03:36.398569 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 13:03:36.398755 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2732,"thread_start_us":817,"threads_started":1}
I20260619 13:03:36.399189 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.386096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.1.87.67:46307)
0619 13:03:36.387703 (+ 1607us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.387713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.388377 (+ 664us) server_negotiation.cc:407] Connection header received
0619 13:03:36.388695 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.388704 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.388925 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.389131 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:36.389855 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.391536 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.395124 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.397719 (+ 2595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.397817 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.398015 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.398719 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.398887 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":1269,"threads_started":1}
I20260619 13:03:36.403838 16187 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad577d80bc74a44b8dcc0fed4ec0967 (DEFAULT_TABLE table=test-table [id=475fb7526d714e548866e0db5da271fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:36.404096 16056 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad577d80bc74a44b8dcc0fed4ec0967 (DEFAULT_TABLE table=test-table [id=475fb7526d714e548866e0db5da271fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:36.405385 16187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad577d80bc74a44b8dcc0fed4ec0967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:36.405385 16056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad577d80bc74a44b8dcc0fed4ec0967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:36.409464 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.389605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:46719 (local address 127.0.0.1:50866)
0619 13:03:36.397153 (+ 7548us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:36.397174 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:36.397275 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:36.401037 (+ 3762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:36.401041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:36.401072 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:36.401248 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.401259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.403539 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.403546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.405332 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.405361 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.408702 (+ 3341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.408710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.408902 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.408939 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:03:36.409030 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 13:03:36.409174 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7441,"thread_start_us":10167,"threads_started":1}
I20260619 13:03:36.409698 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.388642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.1.87.65:46719)
0619 13:03:36.400404 (+ 11762us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.400413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.400439 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:03:36.400540 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.400548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.400810 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.401430 (+ 620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:36.401694 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.403205 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.405578 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.408388 (+ 2810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.408657 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.409038 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.409149 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.409374 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11583,"thread_start_us":11472,"threads_started":1}
I20260619 13:03:36.413566 15924 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad577d80bc74a44b8dcc0fed4ec0967 (DEFAULT_TABLE table=test-table [id=475fb7526d714e548866e0db5da271fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:36.415092 15924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad577d80bc74a44b8dcc0fed4ec0967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:36.419711 16268 tablet_bootstrap.cc:492] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Bootstrap starting.
I20260619 13:03:36.422145 16269 tablet_bootstrap.cc:492] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: Bootstrap starting.
I20260619 13:03:36.424088 16268 tablet_bootstrap.cc:654] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:36.425675 16268 log.cc:824] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:36.425827 16269 tablet_bootstrap.cc:654] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:36.427150 16269 log.cc:824] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:36.428879 16268 tablet_bootstrap.cc:492] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: No bootstrap required, opened a new log
I20260619 13:03:36.429169 16268 ts_tablet_manager.cc:1397] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260619 13:03:36.429685 15989 heartbeater.cc:503] Master 127.1.87.126:37767 was elected leader, sending a full tablet report...
I20260619 13:03:36.430203 16269 tablet_bootstrap.cc:492] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: No bootstrap required, opened a new log
I20260619 13:03:36.430444 16269 ts_tablet_manager.cc:1397] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 13:03:36.438860 16272 tablet_bootstrap.cc:492] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: Bootstrap starting.
I20260619 13:03:36.441701 16268 raft_consensus.cc:348] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:36.442215 16268 raft_consensus.cc:374] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:36.442325 16268 raft_consensus.cc:729] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ce2875ab0b4f2abe705bfb9ef8fbe8, State: Initialized, Role: FOLLOWER
I20260619 13:03:36.442029 16269 raft_consensus.cc:348] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:36.442492 16269 raft_consensus.cc:374] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:36.442633 16269 raft_consensus.cc:729] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669f5a63cb0d40229cd5a59d6d6d31e9, State: Initialized, Role: FOLLOWER
I20260619 13:03:36.443069 16272 tablet_bootstrap.cc:654] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:36.442966 16268 consensus_queue.cc:260] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:36.443253 16269 consensus_queue.cc:260] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:36.444422 16121 heartbeater.cc:503] Master 127.1.87.126:37767 was elected leader, sending a full tablet report...
I20260619 13:03:36.444588 16272 log.cc:824] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:36.445091 16268 ts_tablet_manager.cc:1428] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260619 13:03:36.446144 16252 heartbeater.cc:503] Master 127.1.87.126:37767 was elected leader, sending a full tablet report...
I20260619 13:03:36.447721 16269 ts_tablet_manager.cc:1428] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: Time spent starting tablet: real 0.017s user 0.017s sys 0.001s
I20260619 13:03:36.449400 16272 tablet_bootstrap.cc:492] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: No bootstrap required, opened a new log
I20260619 13:03:36.449719 16272 ts_tablet_manager.cc:1397] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260619 13:03:36.457778 16272 raft_consensus.cc:348] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:36.458228 16272 raft_consensus.cc:374] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:36.458339 16272 raft_consensus.cc:729] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5eda9a7f074f8683585b601e01c573, State: Initialized, Role: FOLLOWER
I20260619 13:03:36.458932 16272 consensus_queue.cc:260] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:36.460707 16272 ts_tablet_manager.cc:1428] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260619 13:03:36.526656 16273 raft_consensus.cc:482] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:03:36.526947 16273 raft_consensus.cc:504] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:36.528733 16273 leader_election.cc:288] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 669f5a63cb0d40229cd5a59d6d6d31e9 (127.1.87.67:46307), 4e5eda9a7f074f8683585b601e01c573 (127.1.87.65:46719)
I20260619 13:03:36.535707 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.529305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:38217 (local address 127.1.87.65:46719)
0619 13:03:36.529532 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.529542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.530294 (+ 752us) server_negotiation.cc:407] Connection header received
0619 13:03:36.530546 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.530551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.530612 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.530718 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:36.531288 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.532034 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.533919 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.534715 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.534872 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.534991 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.535280 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.535433 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:03:36.535887 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.529148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:46719 (local address 127.1.87.66:38217)
0619 13:03:36.530165 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:36.530183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:36.530373 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:36.530747 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:36.530752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:36.530786 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:36.531130 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.531140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.532166 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.532174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.533759 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.533771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.534949 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.534955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.535111 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.535135 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:03:36.535251 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 13:03:36.535425 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":3,"thread_start_us":611,"threads_started":1}
I20260619 13:03:36.536373 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.529150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:46307 (local address 127.1.87.66:59129)
0619 13:03:36.530345 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:36.530360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:36.530517 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:36.530857 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:36.530862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:36.530904 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:36.531382 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.531398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.532382 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.532387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.534039 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:36.534051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.535518 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.535524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:36.535701 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.535728 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 13:03:36.535847 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 13:03:36.535982 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"mutex_wait_us":78,"thread_start_us":717,"threads_started":1}
I20260619 13:03:36.536577 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.529111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:59129 (local address 127.1.87.67:46307)
0619 13:03:36.529358 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.529366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.530452 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 13:03:36.530658 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.530663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.530744 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.530839 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:36.531529 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.532238 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.534191 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.535263 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.535453 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.535560 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.536176 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.536312 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:03:36.536995 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad577d80bc74a44b8dcc0fed4ec0967" candidate_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5eda9a7f074f8683585b601e01c573" is_pre_election: true
I20260619 13:03:36.537415 16207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad577d80bc74a44b8dcc0fed4ec0967" candidate_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" is_pre_election: true
I20260619 13:03:36.537678 15944 raft_consensus.cc:2454] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4ce2875ab0b4f2abe705bfb9ef8fbe8 in term 0.
I20260619 13:03:36.537905 16207 raft_consensus.cc:2454] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4ce2875ab0b4f2abe705bfb9ef8fbe8 in term 0.
I20260619 13:03:36.538504 16011 leader_election.cc:302] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5eda9a7f074f8683585b601e01c573, d4ce2875ab0b4f2abe705bfb9ef8fbe8; no voters:
I20260619 13:03:36.538938 16273 raft_consensus.cc:2781] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:03:36.539103 16273 raft_consensus.cc:482] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:03:36.539227 16273 raft_consensus.cc:3037] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:36.541649 16273 raft_consensus.cc:504] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:36.542356 16273 leader_election.cc:288] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [CANDIDATE]: Term 1 election: Requested vote from peers 669f5a63cb0d40229cd5a59d6d6d31e9 (127.1.87.67:46307), 4e5eda9a7f074f8683585b601e01c573 (127.1.87.65:46719)
I20260619 13:03:36.542968 16207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad577d80bc74a44b8dcc0fed4ec0967" candidate_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9"
I20260619 13:03:36.542971 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad577d80bc74a44b8dcc0fed4ec0967" candidate_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5eda9a7f074f8683585b601e01c573"
I20260619 13:03:36.543227 16207 raft_consensus.cc:3037] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:36.543227 15944 raft_consensus.cc:3037] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:36.546730 16207 raft_consensus.cc:2454] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ce2875ab0b4f2abe705bfb9ef8fbe8 in term 1.
I20260619 13:03:36.546736 15944 raft_consensus.cc:2454] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ce2875ab0b4f2abe705bfb9ef8fbe8 in term 1.
I20260619 13:03:36.547241 16011 leader_election.cc:302] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e5eda9a7f074f8683585b601e01c573, 669f5a63cb0d40229cd5a59d6d6d31e9, d4ce2875ab0b4f2abe705bfb9ef8fbe8; no voters:
I20260619 13:03:36.547636 16273 raft_consensus.cc:2781] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:03:36.548111 16273 raft_consensus.cc:686] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 1 LEADER]: Becoming Leader. State: Replica: d4ce2875ab0b4f2abe705bfb9ef8fbe8, State: Running, Role: LEADER
I20260619 13:03:36.548661 16273 consensus_queue.cc:237] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } }
I20260619 13:03:36.553540 15803 catalog_manager.cc:5495] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4ce2875ab0b4f2abe705bfb9ef8fbe8 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 37845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 } health_report { overall_health: UNKNOWN } } }
W20260619 13:03:36.566892 16253 tablet.cc:2369] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:03:36.592173 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.585642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.1.87.66:37845)
0619 13:03:36.585869 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.585878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.585902 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:03:36.586018 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.586022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.586109 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.586608 (+ 499us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:03:36.587243 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.588100 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.589645 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.590094 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.590250 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.590794 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:03:36.591450 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:03:36.591618 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.591743 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.591843 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:03:36.601513 16207 raft_consensus.cc:1261] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 1 FOLLOWER]: Refusing update from remote peer d4ce2875ab0b4f2abe705bfb9ef8fbe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:36.601513 15944 raft_consensus.cc:1261] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [term 1 FOLLOWER]: Refusing update from remote peer d4ce2875ab0b4f2abe705bfb9ef8fbe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:36.602550 16279 consensus_queue.cc:1035] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "669f5a63cb0d40229cd5a59d6d6d31e9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:03:36.602975 16273 consensus_queue.cc:1035] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e5eda9a7f074f8683585b601e01c573" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 46719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:03:36.606634 16122 tablet.cc:2369] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:03:36.613997 16282 mvcc.cc:205] Tried to move back new op lower bound from 7298556791186874368 to 7298556790987436032. Current Snapshot: MvccSnapshot[applied={T|T < 7298556791186874368}]
I20260619 13:03:36.631383 15803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:03:36.631700 15803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:03:36.635435 15803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Sending DeleteTablet for 3 replicas of tablet 9ad577d80bc74a44b8dcc0fed4ec0967
I20260619 13:03:36.636546 16187 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad577d80bc74a44b8dcc0fed4ec0967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:36 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:50748
I20260619 13:03:36.636688 15924 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad577d80bc74a44b8dcc0fed4ec0967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:36 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:50866
I20260619 13:03:36.636840 16056 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad577d80bc74a44b8dcc0fed4ec0967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:36 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:36498
I20260619 13:03:36.637676 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 15841
I20260619 13:03:36.652043 16292 tablet_replica.cc:330] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: stopping tablet replica
I20260619 13:03:36.652902 16293 tablet_replica.cc:330] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: stopping tablet replica
I20260619 13:03:36.653213 16291 tablet_replica.cc:330] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: stopping tablet replica
I20260619 13:03:36.653703 16292 raft_consensus.cc:2229] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:03:36.653709 16291 raft_consensus.cc:2229] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:03:36.654273 16292 raft_consensus.cc:2258] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:36.654367 16291 raft_consensus.cc:2258] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:36.654610 16293 raft_consensus.cc:2229] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:03:36.655350 16293 raft_consensus.cc:2258] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:36.656612 16291 ts_tablet_manager.cc:1905] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:36.657523 16292 ts_tablet_manager.cc:1905] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:36.663763 16291 ts_tablet_manager.cc:1918] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:36.663949 16291 log.cc:1196] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/9ad577d80bc74a44b8dcc0fed4ec0967
I20260619 13:03:36.664520 16291 ts_tablet_manager.cc:1939] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 669f5a63cb0d40229cd5a59d6d6d31e9: Deleting consensus metadata
I20260619 13:03:36.664986 16292 ts_tablet_manager.cc:1918] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:36.665161 16292 log.cc:1196] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/9ad577d80bc74a44b8dcc0fed4ec0967
I20260619 13:03:36.665887 16292 ts_tablet_manager.cc:1939] T 9ad577d80bc74a44b8dcc0fed4ec0967 P 4e5eda9a7f074f8683585b601e01c573: Deleting consensus metadata
I20260619 13:03:36.666731 15791 catalog_manager.cc:4841] TS 669f5a63cb0d40229cd5a59d6d6d31e9 (127.1.87.67:46307): tablet 9ad577d80bc74a44b8dcc0fed4ec0967 (table test-table [id=475fb7526d714e548866e0db5da271fc]) successfully deleted
I20260619 13:03:36.667840 15790 catalog_manager.cc:4841] TS 4e5eda9a7f074f8683585b601e01c573 (127.1.87.65:46719): tablet 9ad577d80bc74a44b8dcc0fed4ec0967 (table test-table [id=475fb7526d714e548866e0db5da271fc]) successfully deleted
I20260619 13:03:36.668727 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.639625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.1.87.65:46719)
0619 13:03:36.639876 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.639884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.639908 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:03:36.640083 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.640091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.640192 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.640312 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:36.642950 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.644579 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.649147 (+ 4568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.650133 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.650291 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.657759 (+ 7468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:36.657773 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:36.657789 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:36.657835 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:36.661972 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.663045 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.663059 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.663063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.663229 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.665816 (+ 2587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.665822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.665825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.667131 (+ 1306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:36.667487 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.667945 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.668206 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:03:36.668740 16293 ts_tablet_manager.cc:1905] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:36.669416 15984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:36.676951 16293 ts_tablet_manager.cc:1918] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:36.677151 16293 log.cc:1196] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/9ad577d80bc74a44b8dcc0fed4ec0967
I20260619 13:03:36.677750 16293 ts_tablet_manager.cc:1939] T 9ad577d80bc74a44b8dcc0fed4ec0967 P d4ce2875ab0b4f2abe705bfb9ef8fbe8: Deleting consensus metadata
I20260619 13:03:36.679917 15790 catalog_manager.cc:4841] TS d4ce2875ab0b4f2abe705bfb9ef8fbe8 (127.1.87.66:37845): tablet 9ad577d80bc74a44b8dcc0fed4ec0967 (table test-table [id=475fb7526d714e548866e0db5da271fc]) successfully deleted
I20260619 13:03:36.760725 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 15841
I20260619 13:03:36.780054 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 15993
I20260619 13:03:36.801739 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.780907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.1.87.66:37845)
0619 13:03:36.781100 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.781106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.781128 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:03:36.781289 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.781295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.781383 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.781502 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:36.783091 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.784681 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.786043 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.786731 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.786869 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.793990 (+ 7121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:36.794012 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:36.794033 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:36.794088 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:36.798360 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.799172 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.799182 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.799186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.799600 (+ 414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.800057 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.800063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.800066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.800548 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:36.800902 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.801096 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.801342 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:03:36.802580 16116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:36.920248 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 15993
I20260619 13:03:36.943267 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 16124
I20260619 13:03:36.957846 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:36.944621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.1.87.67:46307)
0619 13:03:36.944819 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:36.944823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:36.944839 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:03:36.944892 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:36.944896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:36.944957 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:36.945068 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:36.946350 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.947061 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.948523 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:36.948970 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:36.949109 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:36.951782 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:36.951797 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:36.951814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:36.951863 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:36.954927 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.955743 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.955750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.955752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.955892 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:36.956275 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:36.956284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:36.956286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:36.956629 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:36.956953 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:36.957268 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 13:03:36.957490 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:03:36.958952 16247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:37.054101 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 16124
I20260619 13:03:37.072211 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 15771
I20260619 13:03:37.073431 15834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:37.186870 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 15771
I20260619 13:03:37.209033 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37767
--webserver_interface=127.1.87.126
--webserver_port=33655
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:37767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:37.554132 16298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:37.554445 16298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:37.554541 16298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:37.562333 16298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:37.562469 16298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:37.562515 16298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:37.562567 16298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:03:37.562613 16298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:37.573060 16298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:37767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=33655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:37.574389 16298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:37.575678 16298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:37.588630 16303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:37.588742 16304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:37.588778 16306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:37.589587 16298 server_base.cc:1034] running on GCE node
I20260619 13:03:37.590364 16298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:37.591863 16298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:37.593075 16298 hybrid_clock.cc:648] HybridClock initialized: now 1781874217593022 us; error 56 us; skew 500 ppm
I20260619 13:03:37.593510 16298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:37 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874217, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:37.596763 16298 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:03:37.598479 16298 webserver.cc:460] Webserver started at http://127.1.87.126:33655/ using document root <none> and password file <none>
I20260619 13:03:37.599232 16298 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:37.599413 16298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:37.605026 16298 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:03:37.607869 16313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:37.609107 16298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:37.609303 16298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "8b1dd7ecfcab41d59efe9d9ff552db13"
format_stamp: "Formatted at 2026-06-19 13:03:31 on dist-test-slave-tpsm"
server_key: "4c612900d14f7b168d4408c6d9e73f5c"
server_key_iv: "1871d2454bb6707b9bbf6da9904a7e16"
server_key_version: "encryptionkey@0"
I20260619 13:03:37.610273 16298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:37.630177 16298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:37.635596 16298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:37.635897 16298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:37.654711 16298 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:37767
I20260619 13:03:37.654747 16364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:37767 every 8 connection(s)
I20260619 13:03:37.656384 16298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:03:37.663218 16365 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:03:37.663862 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 16298
I20260619 13:03:37.664851 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:46719
--local_ip_for_outbound_sockets=127.1.87.65
--tserver_master_addrs=127.1.87.126:37767
--webserver_port=41797
--webserver_interface=127.1.87.65
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:03:37.666884 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Bootstrap starting.
I20260619 13:03:37.687747 16365 log.cc:824] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:37.704908 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:03:37.705467 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13: Bootstrap complete.
I20260619 13:03:37.714787 16365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:37.715809 16365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1dd7ecfcab41d59efe9d9ff552db13, State: Initialized, Role: FOLLOWER
I20260619 13:03:37.716478 16365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:37.716713 16365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:03:37.716832 16365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:03:37.717016 16365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:03:37.720981 16365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:37.721494 16365 leader_election.cc:302] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b1dd7ecfcab41d59efe9d9ff552db13; no voters:
I20260619 13:03:37.722157 16365 leader_election.cc:288] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 13:03:37.722306 16369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 13:03:37.723474 16369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [term 3 LEADER]: Becoming Leader. State: Replica: 8b1dd7ecfcab41d59efe9d9ff552db13, State: Running, Role: LEADER
I20260619 13:03:37.724014 16369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } }
I20260619 13:03:37.724383 16365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:03:37.726509 16371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1dd7ecfcab41d59efe9d9ff552db13. Latest consensus state: current_term: 3 leader_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } } }
I20260619 13:03:37.726787 16370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1dd7ecfcab41d59efe9d9ff552db13" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37767 } } }
I20260619 13:03:37.727250 16371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:37.727356 16370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1dd7ecfcab41d59efe9d9ff552db13 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:37.732172 16375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:03:37.742559 16375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:03:37.744529 16375 catalog_manager.cc:1259] Loaded cluster ID: 5f8c3dfdc96449af8fff6d0520d4328a
I20260619 13:03:37.745040 16375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 13:03:37.747501 16375 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb171bd29ad google::LogMessage::Fail() at ??:0
@ 0x7fb171bd588a google::LogMessage::SendToLog() at ??:0
@ 0x7fb171bd24de google::LogMessage::Flush() at ??:0
@ 0x7fb171bd3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb187605512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb187602976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb18768a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb18768a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb17481e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb17482191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb174821799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb1747fceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb177ad66db start_thread at ??:0
@ 0x7fb170a6c61f clone at ??:0
W20260619 13:03:38.071897 16367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:38.072227 16367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:38.072394 16367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:38.080399 16367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:38.080572 16367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:38.080771 16367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:03:38.091207 16367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:46719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=41797
--tserver_master_addrs=127.1.87.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:38.092679 16367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:38.094025 16367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:38.107285 16407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:38.107244 16409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:38.107913 16406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:38.108354 16367 server_base.cc:1034] running on GCE node
I20260619 13:03:38.109040 16367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:38.109951 16367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:38.111220 16367 hybrid_clock.cc:648] HybridClock initialized: now 1781874218111171 us; error 31 us; skew 500 ppm
I20260619 13:03:38.111590 16367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:38 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874218, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:38.114686 16367 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:03:38.116215 16367 webserver.cc:460] Webserver started at http://127.1.87.65:41797/ using document root <none> and password file <none>
I20260619 13:03:38.116892 16367 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:38.116992 16367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:38.122330 16367 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 13:03:38.125011 16416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:38.126397 16367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:03:38.126606 16367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "4e5eda9a7f074f8683585b601e01c573"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "b74989bd4148710dc873e448f6462d8f"
server_key_iv: "7b455d496d45aabde166567927c4e029"
server_key_version: "encryptionkey@0"
I20260619 13:03:38.127671 16367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:38.143942 16367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:38.148304 16367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:38.148597 16367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:38.149734 16367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:38.151175 16367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:38.151304 16367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:38.151472 16367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:38.151571 16367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:38.187341 16367 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:46719
I20260619 13:03:38.187450 16528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:46719 every 8 connection(s)
I20260619 13:03:38.188882 16367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:03:38.190559 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 16367
I20260619 13:03:38.191263 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:37845
--local_ip_for_outbound_sockets=127.1.87.66
--tserver_master_addrs=127.1.87.126:37767
--webserver_port=39855
--webserver_interface=127.1.87.66
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:03:38.196724 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:38.194134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:41055)
0619 13:03:38.195108 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:38.195414 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":358,"threads_started":1}
W20260619 13:03:38.197535 16529 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:37767 (0 consecutive failures): Network error: Failed to ping master at 127.1.87.126:37767: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
I20260619 13:03:38.198619 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:38.198129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:34979)
0619 13:03:38.198335 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:38.198440 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 13:03:38.199575 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:38.199196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:55163)
0619 13:03:38.199376 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:38.199463 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 13:03:38.199875 16529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:03:38.525334 16532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:38.525612 16532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:38.525779 16532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:38.533161 16532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:38.533272 16532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:38.533500 16532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:03:38.544003 16532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:37845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=39855
--tserver_master_addrs=127.1.87.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:38.545337 16532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:38.546694 16532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:38.560485 16540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:38.560603 16537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:38.560572 16538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:38.561568 16532 server_base.cc:1034] running on GCE node
I20260619 13:03:38.562222 16532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:38.563452 16532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:38.564828 16532 hybrid_clock.cc:648] HybridClock initialized: now 1781874218564750 us; error 85 us; skew 500 ppm
I20260619 13:03:38.565176 16532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:38 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874218, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:38.569154 16532 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:03:38.571110 16532 webserver.cc:460] Webserver started at http://127.1.87.66:39855/ using document root <none> and password file <none>
I20260619 13:03:38.571928 16532 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:38.572037 16532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:38.577687 16532 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:03:38.580504 16547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:38.581871 16532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:38.582062 16532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "d4ce2875ab0b4f2abe705bfb9ef8fbe8"
format_stamp: "Formatted at 2026-06-19 13:03:32 on dist-test-slave-tpsm"
server_key: "9dcf9c273103000932f0ece8bfdc847f"
server_key_iv: "16cdfb0850703093d8849e0edad14c7e"
server_key_version: "encryptionkey@0"
I20260619 13:03:38.583089 16532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:38.618160 16532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:38.622737 16532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:38.623024 16532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:38.624094 16532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:38.625475 16532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:38.625594 16532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:38.625738 16532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:38.625826 16532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:38.661134 16532 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:37845
I20260619 13:03:38.661227 16659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:37845 every 8 connection(s)
I20260619 13:03:38.662606 16532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:03:38.667800 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:38.665623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:34155)
0619 13:03:38.666436 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:38.666857 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":343,"threads_started":1}
I20260619 13:03:38.668331 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 16532
W20260619 13:03:38.668457 16660 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:37767 (0 consecutive failures): Network error: Failed to ping master at 127.1.87.126:37767: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
I20260619 13:03:38.669127 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:46307
--local_ip_for_outbound_sockets=127.1.87.67
--tserver_master_addrs=127.1.87.126:37767
--webserver_port=37285
--webserver_interface=127.1.87.67
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:03:38.669467 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:38.669033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:45713)
0619 13:03:38.669187 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:38.669324 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 13:03:38.670316 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:38.670000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:49189)
0619 13:03:38.670144 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:38.670220 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:03:38.670521 16660 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:03:38.996380 16663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:38.996692 16663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:38.996800 16663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:39.004375 16663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:39.004492 16663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:39.004658 16663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:03:39.014537 16663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:46307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=37285
--tserver_master_addrs=127.1.87.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:39.015880 16663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:39.017156 16663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:39.030045 16668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:39.030085 16669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:39.030128 16671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:39.030962 16663 server_base.cc:1034] running on GCE node
I20260619 13:03:39.031702 16663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:39.032709 16663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:39.033957 16663 hybrid_clock.cc:648] HybridClock initialized: now 1781874219033898 us; error 44 us; skew 500 ppm
I20260619 13:03:39.034317 16663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:39 dist-test-slave-tpsm krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874219, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:39.037658 16663 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:03:39.039247 16663 webserver.cc:460] Webserver started at http://127.1.87.67:37285/ using document root <none> and password file <none>
I20260619 13:03:39.040088 16663 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:39.040237 16663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:39.045476 16663 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 13:03:39.048213 16678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:39.049520 16663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:39.049706 16663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "669f5a63cb0d40229cd5a59d6d6d31e9"
format_stamp: "Formatted at 2026-06-19 13:03:33 on dist-test-slave-tpsm"
server_key: "ff327b6aa9c62f5c2828df7077378b71"
server_key_iv: "f0a0ce0c482153e5e113b2446d431ab3"
server_key_version: "encryptionkey@0"
I20260619 13:03:39.050659 16663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:39.072901 16663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:39.077608 16663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:39.077903 16663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:39.078965 16663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:39.080431 16663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:39.080570 16663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:39.080703 16663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:39.080792 16663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:39.114127 16663 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:46307
I20260619 13:03:39.114216 16790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:46307 every 8 connection(s)
I20260619 13:03:39.115597 16663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:03:39.120016 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:39.118297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:37977)
0619 13:03:39.118967 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:39.119160 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
W20260619 13:03:39.120615 16791 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:37767 (0 consecutive failures): Network error: Failed to ping master at 127.1.87.126:37767: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
I20260619 13:03:39.121337 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:39.121059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:57699)
0619 13:03:39.121182 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:39.121242 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 13:03:39.122071 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:39.121800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:37651)
0619 13:03:39.121909 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:39.121975 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 13:03:39.122294 16791 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 13:03:39.122845 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 16663
I20260619 13:03:39.201272 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:39.200473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:45187)
0619 13:03:39.200922 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:39.201028 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":225,"threads_started":1}
I20260619 13:03:39.671959 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:39.671082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:57803)
0619 13:03:39.671608 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:39.671717 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":304,"threads_started":1}
I20260619 13:03:40.123780 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:40.122841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:60099)
0619 13:03:40.123295 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:40.123551 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260619 13:03:40.203176 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:40.202448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:56675)
0619 13:03:40.202911 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:40.202994 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":232,"threads_started":1}
I20260619 13:03:40.674113 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:40.673195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:39229)
0619 13:03:40.673781 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:40.673889 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":245,"threads_started":1}
I20260619 13:03:41.125569 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:41.124830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:50579)
0619 13:03:41.125310 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:41.125393 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
I20260619 13:03:41.205098 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:41.204306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:40539)
0619 13:03:41.204783 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:41.204864 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":260,"threads_started":1}
I20260619 13:03:41.675819 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:41.675069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:34635)
0619 13:03:41.675563 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:41.675648 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":298,"threads_started":1}
I20260619 13:03:42.127540 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:42.126574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:35789)
0619 13:03:42.127139 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:42.127230 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":287,"threads_started":1}
I20260619 13:03:42.206892 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:42.206116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:52661)
0619 13:03:42.206594 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:42.206695 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260619 13:03:42.677346 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:42.676654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:46853)
0619 13:03:42.677091 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:42.677171 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260619 13:03:43.129117 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:43.128412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:54817)
0619 13:03:43.128841 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:43.128916 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":229,"threads_started":1}
I20260619 13:03:43.208878 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:43.207944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:46023)
0619 13:03:43.208570 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:43.208658 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":338,"threads_started":1}
I20260619 13:03:43.679093 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:43.678372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:40241)
0619 13:03:43.678829 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:43.678912 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260619 13:03:44.130786 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:44.130038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:47675)
0619 13:03:44.130511 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:44.130606 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":255,"threads_started":1}
I20260619 13:03:44.210533 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:44.209819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:35271)
0619 13:03:44.210275 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:44.210364 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260619 13:03:44.681106 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:44.680139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:40325)
0619 13:03:44.680819 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:44.680916 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":479,"threads_started":1}
I20260619 13:03:45.132717 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:45.131911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:34709)
0619 13:03:45.132368 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:45.132458 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":245,"threads_started":1}
I20260619 13:03:45.212483 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:45.211627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:42955)
0619 13:03:45.212159 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:45.212243 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":252,"threads_started":1}
I20260619 13:03:45.683919 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:45.682560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:52095)
0619 13:03:45.683241 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:45.683580 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":301,"threads_started":1}
I20260619 13:03:46.134616 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:46.133805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:51033)
0619 13:03:46.134297 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:46.134382 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":215,"threads_started":1}
I20260619 13:03:46.214295 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:46.213531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:43153)
0619 13:03:46.213997 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:46.214083 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":251,"threads_started":1}
I20260619 13:03:46.685737 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:46.685018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:36781)
0619 13:03:46.685469 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:46.685556 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260619 13:03:47.136828 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:47.135779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:56441)
0619 13:03:47.136414 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:47.136641 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":296,"threads_started":1}
I20260619 13:03:47.216069 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:47.215296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:43141)
0619 13:03:47.215816 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:47.215896 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":235,"threads_started":1}
I20260619 13:03:47.687881 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:47.686781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:39535)
0619 13:03:47.687554 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:47.687708 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":592,"threads_started":1}
I20260619 13:03:48.138659 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:48.137886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:57171)
0619 13:03:48.138394 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:48.138480 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
I20260619 13:03:48.218000 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:48.217219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:49949)
0619 13:03:48.217749 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:48.217837 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":287,"threads_started":1}
I20260619 13:03:48.689754 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:48.688888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:39929)
0619 13:03:48.689451 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:48.689533 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":333,"threads_started":1}
I20260619 13:03:49.140575 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:49.139858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:56345)
0619 13:03:49.140329 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:49.140413 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":262,"threads_started":1}
I20260619 13:03:49.219807 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:49.218997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:43045)
0619 13:03:49.219490 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:49.219622 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":225,"threads_started":1}
I20260619 13:03:49.691430 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:49.690677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:53757)
0619 13:03:49.691093 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:49.691182 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":213,"threads_started":1}
I20260619 13:03:50.142241 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:50.141512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:41381)
0619 13:03:50.141956 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:50.142049 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":220,"threads_started":1}
I20260619 13:03:50.221552 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:50.220814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:54767)
0619 13:03:50.221295 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:50.221392 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":258,"threads_started":1}
I20260619 13:03:50.693184 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:50.692403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:50897)
0619 13:03:50.692944 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:50.693027 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":310,"threads_started":1}
I20260619 13:03:51.143957 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:51.143241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:43731)
0619 13:03:51.143724 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:51.143802 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
I20260619 13:03:51.223415 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:51.222487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:36547)
0619 13:03:51.222991 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:51.223119 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":255,"threads_started":1}
I20260619 13:03:51.695076 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:51.694202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:52565)
0619 13:03:51.694768 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:51.694889 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":310,"threads_started":1}
I20260619 13:03:52.145745 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:52.144945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:52449)
0619 13:03:52.145438 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:52.145544 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":244,"threads_started":1}
I20260619 13:03:52.225479 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:52.224623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:57863)
0619 13:03:52.225100 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:52.225194 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":237,"threads_started":1}
I20260619 13:03:52.696858 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:52.695997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:60009)
0619 13:03:52.696580 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:52.696672 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":320,"threads_started":1}
I20260619 13:03:53.148118 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:53.147154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:57675)
0619 13:03:53.147774 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:53.147892 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":366,"threads_started":1}
I20260619 13:03:53.227644 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:53.226610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:36441)
0619 13:03:53.227091 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:53.227191 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":277,"threads_started":1}
I20260619 13:03:53.699153 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:53.698018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.66:47839)
0619 13:03:53.698742 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:53.698891 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":213,"threads_started":1}
I20260619 13:03:54.134445 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 16367
I20260619 13:03:54.150312 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:54.149508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.67:45837)
0619 13:03:54.150027 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:54.150124 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":288,"threads_started":1}
I20260619 13:03:54.154006 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:54.135823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.1.87.65:46719)
0619 13:03:54.136600 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:54.136631 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:54.136679 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:03:54.138651 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:54.138666 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:54.139119 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:54.139505 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:54.141090 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:54.142637 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:54.143946 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:54.144618 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:54.144818 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:54.147486 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:54.147494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:54.147510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:54.147552 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:54.150597 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:54.151301 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:54.151349 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:54.151359 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:54.151542 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:54.151952 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:54.151957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:54.151960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:54.152503 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:54.152830 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:54.153248 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 13:03:54.153721 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":402,"threads_started":1}
I20260619 13:03:54.155975 16524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:54.241469 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:54.239873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37767 (local address 127.1.87.65:33915)
0619 13:03:54.240566 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:54.240796 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":388,"threads_started":1}
I20260619 13:03:54.243624 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 16367
I20260619 13:03:54.262818 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 16532
I20260619 13:03:54.282881 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:54.263884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.1.87.66:37845)
0619 13:03:54.264418 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:54.264432 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:54.264469 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:03:54.265954 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:54.265963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:54.266520 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:54.267122 (+ 602us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:54.269801 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:54.271083 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:54.272649 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:54.273373 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:54.273536 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:54.276467 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:54.276475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:54.276491 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:54.276535 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:54.279676 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:54.280415 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:54.280422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:54.280428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:54.280597 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:54.280933 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:54.280936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:54.280938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:54.281488 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:54.281724 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:54.282132 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 13:03:54.282579 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":251,"threads_started":1}
I20260619 13:03:54.284493 16655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:54.375483 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 16532
I20260619 13:03:54.393929 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 16663
I20260619 13:03:54.413175 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:54.394993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.1.87.67:46307)
0619 13:03:54.395647 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:54.395661 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:54.395702 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:03:54.397336 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:54.397345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:54.397784 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:54.398067 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:54.399723 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:54.401071 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:54.402480 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:54.403834 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:54.404059 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:54.406762 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:54.406771 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:54.406787 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:54.406835 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:54.410095 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:54.410774 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:54.410780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:54.410787 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:54.410945 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:54.411268 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:54.411271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:54.411273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:54.411873 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:54.412079 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:54.412445 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 13:03:54.412877 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":343,"threads_started":1}
I20260619 13:03:54.414539 16786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:54.501991 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 16663
2026-06-19T13:03:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24600 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:03:54 dist-test-slave-tpsm krb5kdc[16849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:03:54 dist-test-slave-tpsm krb5kdc[16849](info): set up 2 sockets
Jun 19 13:03:54 dist-test-slave-tpsm krb5kdc[16849](info): commencing operation
krb5kdc: starting...
W20260619 13:03:55.624128 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:03:55 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:03:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:03:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:03:55.875093 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:33205
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:56.215807 16862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:56.216185 16862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:56.216372 16862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:56.223682 16862 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 13:03:56.223829 16862 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 13:03:56.223897 16862 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 13:03:56.223971 16862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:03:56.224049 16862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:56.224148 16862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:56.224217 16862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:03:56.224299 16862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:03:56.234177 16862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:33205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:56.235598 16862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:56.236905 16862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:56.249554 16867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:56.249629 16870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:56.249581 16868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:56.250612 16862 server_base.cc:1034] running on GCE node
I20260619 13:03:56.251499 16862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:56.253094 16862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:56.254349 16862 hybrid_clock.cc:648] HybridClock initialized: now 1781874236254272 us; error 54 us; skew 500 ppm
I20260619 13:03:56.254722 16862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:56 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874236, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:56.258303 16862 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:03:56.260176 16862 webserver.cc:460] Webserver started at http://127.1.87.126:44323/ using document root <none> and password file <none>
I20260619 13:03:56.260883 16862 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:56.260973 16862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:56.261236 16862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:56.265249 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "c4450e69c2a340db8416618432ad8b82"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "d9f34208c98f134be88562eb495e659e"
server_key_iv: "9d7e0a4c09877be253f3058c7b7d3d3e"
server_key_version: "encryptionkey@0"
I20260619 13:03:56.266431 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "c4450e69c2a340db8416618432ad8b82"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "d9f34208c98f134be88562eb495e659e"
server_key_iv: "9d7e0a4c09877be253f3058c7b7d3d3e"
server_key_version: "encryptionkey@0"
I20260619 13:03:56.272711 16862 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260619 13:03:56.276163 16877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:56.277544 16862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:03:56.277734 16862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "c4450e69c2a340db8416618432ad8b82"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "d9f34208c98f134be88562eb495e659e"
server_key_iv: "9d7e0a4c09877be253f3058c7b7d3d3e"
server_key_version: "encryptionkey@0"
I20260619 13:03:56.277961 16862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:56.311792 16862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:56.316000 16862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:56.316223 16862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:56.336074 16862 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:33205
I20260619 13:03:56.336107 16928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:33205 every 8 connection(s)
I20260619 13:03:56.337451 16862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:03:56.339778 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 16862
I20260619 13:03:56.339977 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:03:56.340675 1373 external_mini_cluster.cc:1428] Setting key f3d96822e3a53961c2af48c163744fb4
I20260619 13:03:56.341461 16929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:56.356117 16929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Bootstrap starting.
Jun 19 13:03:56 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:56.360020 16929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:56.361402 16929 log.cc:824] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:56.365097 16929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: No bootstrap required, opened a new log
I20260619 13:03:56.366706 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:56.342604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.1.87.126:33205)
0619 13:03:56.343473 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:56.343484 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:56.343524 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:03:56.345940 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:56.345963 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:56.346345 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:56.346688 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:56.348515 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:56.349951 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:56.351877 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:56.352649 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:56.352807 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:56.357798 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:56.357812 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:56.357831 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:56.357893 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:56.361855 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:56.362817 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:56.362824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:56.362829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:56.363098 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:56.363566 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:56.363571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:56.363574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:56.364270 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:56.364593 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:56.364939 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 13:03:56.365379 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":379,"threads_started":1}
I20260619 13:03:56.373694 16929 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:03:56.374096 16929 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:56.374174 16929 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4450e69c2a340db8416618432ad8b82, State: Initialized, Role: FOLLOWER
I20260619 13:03:56.374694 16929 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:03:56.374886 16929 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:03:56.374977 16929 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:03:56.375092 16929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:56.377249 16929 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:03:56.377707 16929 leader_election.cc:302] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4450e69c2a340db8416618432ad8b82; no voters:
I20260619 13:03:56.378232 16929 leader_election.cc:288] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:03:56.378400 16934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:03:56.379105 16934 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 1 LEADER]: Becoming Leader. State: Replica: c4450e69c2a340db8416618432ad8b82, State: Running, Role: LEADER
I20260619 13:03:56.379705 16934 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:03:56.380334 16929 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:03:56.382876 16936 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4450e69c2a340db8416618432ad8b82. Latest consensus state: current_term: 1 leader_uuid: "c4450e69c2a340db8416618432ad8b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } } }
I20260619 13:03:56.383889 16936 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:56.384049 16935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4450e69c2a340db8416618432ad8b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } } }
I20260619 13:03:56.384456 16935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: This master's current role is: LEADER
I20260619 13:03:56.385265 16942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:03:56.390699 16942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:03:56.399741 16942 catalog_manager.cc:1347] Generated new cluster ID: b980185d8bcc4567900d39aa9948155a
I20260619 13:03:56.399830 16942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:03:56.411657 16942 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:03:56.412976 16942 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:03:56.421777 16942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Generated new TSK 0
I20260619 13:03:56.422760 16942 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:03:56.489094 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33205
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:56.822526 16955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:56.822813 16955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:56.822996 16955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:56.830848 16955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:56.830986 16955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:56.831171 16955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:03:56.842077 16955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:56.843456 16955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:56.844907 16955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:56.858064 16960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:56.858115 16961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:56.858038 16963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:56.858960 16955 server_base.cc:1034] running on GCE node
I20260619 13:03:56.859702 16955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:56.860823 16955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:56.862046 16955 hybrid_clock.cc:648] HybridClock initialized: now 1781874236861990 us; error 44 us; skew 500 ppm
I20260619 13:03:56.862354 16955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:56 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874236, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:56.865554 16955 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:03:56.867101 16955 webserver.cc:460] Webserver started at http://127.1.87.65:37031/ using document root <none> and password file <none>
I20260619 13:03:56.867890 16955 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:56.868042 16955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:56.868384 16955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:56.871501 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "4d7501ee179c45e5bfcf4c199d764f2e"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "841f18d81e351a86e26227b84a988c53"
server_key_iv: "47774dd07afdb8a983dc96ad1deb89b5"
server_key_version: "encryptionkey@0"
I20260619 13:03:56.872313 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "4d7501ee179c45e5bfcf4c199d764f2e"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "841f18d81e351a86e26227b84a988c53"
server_key_iv: "47774dd07afdb8a983dc96ad1deb89b5"
server_key_version: "encryptionkey@0"
I20260619 13:03:56.877570 16955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:03:56.881261 16970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:56.882866 16955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:03:56.883061 16955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "4d7501ee179c45e5bfcf4c199d764f2e"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "841f18d81e351a86e26227b84a988c53"
server_key_iv: "47774dd07afdb8a983dc96ad1deb89b5"
server_key_version: "encryptionkey@0"
I20260619 13:03:56.883275 16955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:56.902228 16955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:56.906742 16955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:56.907061 16955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:56.908115 16955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:56.909873 16955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:56.910060 16955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:56.910259 16955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:56.910322 16955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:56.947587 16955 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:39873
I20260619 13:03:56.947649 17082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:39873 every 8 connection(s)
I20260619 13:03:56.948967 16955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:03:56.955142 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 16955
I20260619 13:03:56.955381 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:03:56.956104 1373 external_mini_cluster.cc:1428] Setting key ae3532f2341f30acc8480d9260b2a679
Jun 19 13:03:56 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874236, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:56.970659 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:56.951902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:41217 (local address 127.1.87.126:33205)
0619 13:03:56.952371 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:56.952375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:56.953346 (+ 971us) server_negotiation.cc:407] Connection header received
0619 13:03:56.955606 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:56.955612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:56.955718 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:56.955846 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:56.957618 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:56.958426 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:56.960063 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:56.960457 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:56.960609 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:56.964737 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:56.964743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:56.964746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:56.964780 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:56.967401 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:56.968135 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:56.968141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:56.968145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:56.968270 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:56.968701 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:56.968706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:56.968709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:56.968981 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:56.969137 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:56.969854 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 13:03:56.970077 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
I20260619 13:03:56.971904 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:56.952424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:41217)
0619 13:03:56.953141 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:56.953216 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:56.954884 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:56.956123 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:56.956132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:56.956719 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:56.957439 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:56.957457 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:56.958598 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:56.958602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:56.959914 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:56.959926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:56.960689 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:56.960698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:56.960941 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:56.961975 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:56.962005 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:56.964464 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:56.967545 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:56.967553 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:56.967571 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:56.968002 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:56.968386 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:56.968390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:56.968394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:56.968581 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:56.969109 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:56.969117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:56.969481 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 13:03:56.969755 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 13:03:56.970279 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":290,"threads_started":1}
I20260619 13:03:56.973606 17083 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33205
I20260619 13:03:56.973966 17083 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:56.974934 17083 heartbeater.cc:511] Master 127.1.87.126:33205 requested a full tablet report, sending...
I20260619 13:03:56.977478 16894 ts_manager.cc:194] Registered new tserver with Master: 4d7501ee179c45e5bfcf4c199d764f2e (127.1.87.65:39873)
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
I20260619 13:03:56.978992 16894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:41217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:03:57.007091 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33205
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:57.335440 17089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:57.335992 17089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:57.336174 17089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:57.344065 17089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:57.344244 17089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:57.344427 17089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:03:57.354857 17089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:57.356324 17089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:57.357671 17089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:57.372008 17097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:57.372112 17094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:57.372136 17095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:57.372972 17089 server_base.cc:1034] running on GCE node
I20260619 13:03:57.373590 17089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:57.374573 17089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:57.375798 17089 hybrid_clock.cc:648] HybridClock initialized: now 1781874237375772 us; error 62 us; skew 500 ppm
I20260619 13:03:57.376116 17089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:57 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:57.379271 17089 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:03:57.381011 17089 webserver.cc:460] Webserver started at http://127.1.87.66:35915/ using document root <none> and password file <none>
I20260619 13:03:57.381832 17089 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:57.381937 17089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:57.382267 17089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:57.385363 17089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "b089d29fda264c4a88cdb082163ac8d4"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "46e5eb33e158988733c7e28dd2020ce5"
server_key_iv: "d8e986b603b1ff583b4244d03495d851"
server_key_version: "encryptionkey@0"
I20260619 13:03:57.386144 17089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "b089d29fda264c4a88cdb082163ac8d4"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "46e5eb33e158988733c7e28dd2020ce5"
server_key_iv: "d8e986b603b1ff583b4244d03495d851"
server_key_version: "encryptionkey@0"
I20260619 13:03:57.391635 17089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:03:57.395251 17104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:57.396708 17089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:57.396898 17089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "b089d29fda264c4a88cdb082163ac8d4"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "46e5eb33e158988733c7e28dd2020ce5"
server_key_iv: "d8e986b603b1ff583b4244d03495d851"
server_key_version: "encryptionkey@0"
I20260619 13:03:57.397114 17089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:57.416790 17089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:57.420982 17089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:57.421314 17089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:57.422317 17089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:57.423731 17089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:57.423836 17089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:57.423990 17089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:57.424064 17089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:57.461906 17089 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:38229
I20260619 13:03:57.462064 17216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:38229 every 8 connection(s)
I20260619 13:03:57.463264 17089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:03:57.472148 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 17089
I20260619 13:03:57.472376 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:03:57.472929 1373 external_mini_cluster.cc:1428] Setting key 6ccfc119cb72b2ad19edc8a7f82826cf
Jun 19 13:03:57 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:57.485569 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:57.465846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:41461 (local address 127.1.87.126:33205)
0619 13:03:57.466057 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:57.466064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:57.467350 (+ 1286us) server_negotiation.cc:407] Connection header received
0619 13:03:57.469108 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:57.469113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:57.469182 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:57.469287 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:57.470726 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:57.471518 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:57.473784 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:57.474380 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:57.474511 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:57.479384 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:57.479390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:57.479394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:57.479430 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:57.482338 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:57.483126 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:57.483132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:57.483137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:57.483308 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:57.483779 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:57.483785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:57.483789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:57.484065 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:57.484214 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:57.484925 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 13:03:57.485157 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:03:57.486603 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:57.466447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:41461)
0619 13:03:57.467147 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:57.467192 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:57.468857 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:57.469488 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:57.469496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:57.470009 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:57.470549 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:57.470565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:57.471650 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:57.471653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:57.473629 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:57.473642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:57.474749 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:57.474757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:57.475301 (+ 544us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:57.476131 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:57.476175 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:57.479073 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:57.482493 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:57.482503 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:57.482524 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:57.482988 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:57.483454 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:57.483459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:57.483463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:57.483662 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:57.484209 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:57.484219 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:57.484568 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 13:03:57.484848 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 13:03:57.485409 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":282,"threads_started":1}
I20260619 13:03:57.488080 17217 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33205
I20260619 13:03:57.488408 17217 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:57.489137 17217 heartbeater.cc:511] Master 127.1.87.126:33205 requested a full tablet report, sending...
I20260619 13:03:57.490767 16894 ts_manager.cc:194] Registered new tserver with Master: b089d29fda264c4a88cdb082163ac8d4 (127.1.87.66:38229)
I20260619 13:03:57.492082 16894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:41461
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:03:57.527056 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33205
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:03:57.869184 17222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:03:57.869562 17222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:03:57.869664 17222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:03:57.877067 17222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:03:57.877178 17222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:03:57.877290 17222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:03:57.887737 17222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:03:57.888978 17222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:03:57.890733 17222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:03:57.904052 17228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:57.904150 17230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:03:57.904594 17227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:03:57.904852 17222 server_base.cc:1034] running on GCE node
I20260619 13:03:57.905454 17222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:03:57.906396 17222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:03:57.907605 17222 hybrid_clock.cc:648] HybridClock initialized: now 1781874237907553 us; error 39 us; skew 500 ppm
I20260619 13:03:57.907929 17222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:03:57 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:03:57.911309 17222 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:03:57.912858 17222 webserver.cc:460] Webserver started at http://127.1.87.67:46451/ using document root <none> and password file <none>
I20260619 13:03:57.913542 17222 fs_manager.cc:359] Metadata directory not provided
I20260619 13:03:57.913687 17222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:03:57.914000 17222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:03:57.916919 17222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "b3464efc4ba74b31b68c23dd72959ad9"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "b46ad94f393166f7749877c23d1f7e8c"
server_key_iv: "1e3e42f28cff1afa5e605ec77798b9bd"
server_key_version: "encryptionkey@0"
I20260619 13:03:57.917663 17222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "b3464efc4ba74b31b68c23dd72959ad9"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "b46ad94f393166f7749877c23d1f7e8c"
server_key_iv: "1e3e42f28cff1afa5e605ec77798b9bd"
server_key_version: "encryptionkey@0"
I20260619 13:03:57.922830 17222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 13:03:57.926304 17237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:57.927731 17222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:03:57.927901 17222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "b3464efc4ba74b31b68c23dd72959ad9"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "b46ad94f393166f7749877c23d1f7e8c"
server_key_iv: "1e3e42f28cff1afa5e605ec77798b9bd"
server_key_version: "encryptionkey@0"
I20260619 13:03:57.928097 17222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:03:57.969978 17222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:03:57.974578 17222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:03:57.974920 17222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:03:57.976047 17222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:03:57.977459 17222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:03:57.977561 17222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:57.977715 17222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:03:57.977788 17222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:03:57.982679 17083 heartbeater.cc:503] Master 127.1.87.126:33205 was elected leader, sending a full tablet report...
I20260619 13:03:58.012122 17222 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:36423
I20260619 13:03:58.012158 17349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:36423 every 8 connection(s)
I20260619 13:03:58.013388 17222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:03:58.022451 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 17222
I20260619 13:03:58.022675 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:03:58.023126 1373 external_mini_cluster.cc:1428] Setting key 9e40f365131b4cdd5eb25de8173554a6
Jun 19 13:03:58 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874237, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:03:58.033310 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.016011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:45371 (local address 127.1.87.126:33205)
0619 13:03:58.016501 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.016506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.017284 (+ 778us) server_negotiation.cc:407] Connection header received
0619 13:03:58.019111 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.019117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.019214 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.019355 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:58.020833 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.021619 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.023240 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.023754 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.023893 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.027668 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:58.027673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:58.027676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:58.027711 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:58.030412 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.031135 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.031140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.031144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.031277 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.031757 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.031760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.031762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.032070 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:58.032235 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.032860 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.033020 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":275,"threads_started":1}
I20260619 13:03:58.034289 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.016466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:45371)
0619 13:03:58.017128 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:58.017170 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:58.018847 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:58.019588 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:58.019596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:58.020100 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:03:58.020636 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.020652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.021780 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.021784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.023092 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.023104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.024041 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.024049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.024315 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.025063 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:03:58.025097 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:03:58.027486 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:03:58.030539 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:58.030546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:58.030558 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:58.030998 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:58.031481 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:03:58.031485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:03:58.031487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:03:58.031658 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:03:58.032197 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:03:58.032202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:03:58.032516 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 13:03:58.032780 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 13:03:58.033159 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":274,"threads_started":1}
I20260619 13:03:58.035692 17350 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33205
I20260619 13:03:58.035995 17350 heartbeater.cc:464] Registering TS with master...
I20260619 13:03:58.036732 17350 heartbeater.cc:511] Master 127.1.87.126:33205 requested a full tablet report, sending...
I20260619 13:03:58.038278 16894 ts_manager.cc:194] Registered new tserver with Master: b3464efc4ba74b31b68c23dd72959ad9 (127.1.87.67:36423)
I20260619 13:03:58.039117 16894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:45371
I20260619 13:03:58.050971 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:03:58.069422 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.055566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.1.87.126:33205)
0619 13:03:58.055732 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.055736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.056280 (+ 544us) server_negotiation.cc:407] Connection header received
0619 13:03:58.056480 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.056484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.056567 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.056661 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:58.058347 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.059072 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.060638 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.061059 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.061163 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.063925 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:58.063931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:58.063934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:58.063970 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:58.066640 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.067406 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.067410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.067412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.067567 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.067906 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.067909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.067911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.068239 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:58.068420 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.068994 (+ 574us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.069170 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:03:58.073904 16894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:03:58.081189 16894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:03:58.115424 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.095639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.1.87.66:38229)
0619 13:03:58.105933 (+ 10294us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.105944 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.107131 (+ 1187us) server_negotiation.cc:407] Connection header received
0619 13:03:58.108242 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.108254 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.108496 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.108671 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:58.109250 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.110785 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.113260 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.114197 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.114351 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.114570 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.114875 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.115041 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10083,"thread_start_us":10853,"threads_started":1}
I20260619 13:03:58.115566 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.094744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36423 (local address 127.0.0.1:45038)
0619 13:03:58.106169 (+ 11425us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:58.106226 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:58.106437 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:58.107352 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:58.107361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:58.107399 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:58.107873 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.107886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.109961 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.109966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.112517 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.112535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.114743 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.114750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.114916 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.114953 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:03:58.115019 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 13:03:58.115160 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11033,"thread_start_us":11997,"threads_started":1}
I20260619 13:03:58.116173 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.095030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.1.87.67:36423)
0619 13:03:58.106612 (+ 11582us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.106620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.106668 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:03:58.106760 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.106770 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.106989 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.107217 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:58.108056 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.109832 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.112855 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.114494 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.115276 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.115524 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.115641 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.115799 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11400,"thread_start_us":11547,"threads_started":1}
I20260619 13:03:58.116784 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.095042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:38229 (local address 127.0.0.1:60848)
0619 13:03:58.107024 (+ 11982us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:58.107042 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:58.107985 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:58.108750 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:58.108754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:58.108779 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:58.109078 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.109086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.110948 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.110953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.113113 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.113124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.114427 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.114434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.114654 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.114692 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:03:58.114834 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 13:03:58.115024 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11844,"mutex_wait_us":1,"thread_start_us":10921,"threads_started":1}
I20260619 13:03:58.117444 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.107149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:59538)
0619 13:03:58.108619 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:58.108638 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:58.108764 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:58.109392 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:58.109398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:58.109422 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:58.109787 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.109798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.111744 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.111749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.114458 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.114474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.116279 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.116286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.116455 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.116491 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:03:58.116885 (+ 394us) client_negotiation.cc:240] Negotiation successful
0619 13:03:58.117038 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1351,"mutex_wait_us":1,"thread_start_us":1200,"threads_started":1}
I20260619 13:03:58.118108 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.107425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.1.87.65:39873)
0619 13:03:58.108628 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.108635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.108888 (+ 253us) server_negotiation.cc:407] Connection header received
0619 13:03:58.108978 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.108984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.109185 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.109379 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:58.109952 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.111583 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.114663 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.116068 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.116234 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.116448 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.117633 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.117817 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":288,"threads_started":1}
I20260619 13:03:58.120203 17285 tablet_service.cc:1467] Processing CreateTablet for tablet ce1f99d6ffda40e2a8bb743de57f029a (DEFAULT_TABLE table=test-table [id=1757acb7fb574f91b0cd214e4fefa7c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:58.120456 17152 tablet_service.cc:1467] Processing CreateTablet for tablet ce1f99d6ffda40e2a8bb743de57f029a (DEFAULT_TABLE table=test-table [id=1757acb7fb574f91b0cd214e4fefa7c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:58.121491 17285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1f99d6ffda40e2a8bb743de57f029a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:58.121965 17152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1f99d6ffda40e2a8bb743de57f029a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:58.121367 17018 tablet_service.cc:1467] Processing CreateTablet for tablet ce1f99d6ffda40e2a8bb743de57f029a (DEFAULT_TABLE table=test-table [id=1757acb7fb574f91b0cd214e4fefa7c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:03:58.122844 17018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1f99d6ffda40e2a8bb743de57f029a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:03:58.135280 17369 tablet_bootstrap.cc:492] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: Bootstrap starting.
I20260619 13:03:58.138433 17370 tablet_bootstrap.cc:492] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: Bootstrap starting.
I20260619 13:03:58.138881 17369 tablet_bootstrap.cc:654] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:58.140322 17369 log.cc:824] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:58.141899 17371 tablet_bootstrap.cc:492] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: Bootstrap starting.
I20260619 13:03:58.142431 17370 tablet_bootstrap.cc:654] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:58.143848 17370 log.cc:824] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:58.144078 17369 tablet_bootstrap.cc:492] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: No bootstrap required, opened a new log
I20260619 13:03:58.144384 17369 ts_tablet_manager.cc:1397] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 13:03:58.145907 17371 tablet_bootstrap.cc:654] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: Neither blocks nor log segments found. Creating new log.
I20260619 13:03:58.147042 17371 log.cc:824] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: Log is configured to *not* fsync() on all Append() calls
I20260619 13:03:58.147162 17370 tablet_bootstrap.cc:492] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: No bootstrap required, opened a new log
I20260619 13:03:58.147506 17370 ts_tablet_manager.cc:1397] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:03:58.149673 17371 tablet_bootstrap.cc:492] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: No bootstrap required, opened a new log
I20260619 13:03:58.149953 17371 ts_tablet_manager.cc:1397] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:03:58.155742 17369 raft_consensus.cc:348] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:03:58.156250 17369 raft_consensus.cc:374] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:58.156420 17369 raft_consensus.cc:729] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3464efc4ba74b31b68c23dd72959ad9, State: Initialized, Role: FOLLOWER
I20260619 13:03:58.157150 17369 consensus_queue.cc:260] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:03:58.158468 17371 raft_consensus.cc:348] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:03:58.158509 17370 raft_consensus.cc:348] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:03:58.158932 17370 raft_consensus.cc:374] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:58.158952 17371 raft_consensus.cc:374] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:03:58.159050 17371 raft_consensus.cc:729] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b089d29fda264c4a88cdb082163ac8d4, State: Initialized, Role: FOLLOWER
I20260619 13:03:58.159090 17369 ts_tablet_manager.cc:1428] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 13:03:58.159121 17370 raft_consensus.cc:729] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7501ee179c45e5bfcf4c199d764f2e, State: Initialized, Role: FOLLOWER
I20260619 13:03:58.159761 17350 heartbeater.cc:503] Master 127.1.87.126:33205 was elected leader, sending a full tablet report...
I20260619 13:03:58.159713 17371 consensus_queue.cc:260] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:03:58.159873 17370 consensus_queue.cc:260] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:03:58.160771 17217 heartbeater.cc:503] Master 127.1.87.126:33205 was elected leader, sending a full tablet report...
I20260619 13:03:58.161422 17371 ts_tablet_manager.cc:1428] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 13:03:58.163007 17370 ts_tablet_manager.cc:1428] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260619 13:03:58.201740 17084 tablet.cc:2369] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:03:58.215741 17218 tablet.cc:2369] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:03:58.265174 17351 tablet.cc:2369] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:03:58.505409 17375 raft_consensus.cc:482] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:03:58.505679 17375 raft_consensus.cc:504] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:03:58.507241 17375 leader_election.cc:288] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b089d29fda264c4a88cdb082163ac8d4 (127.1.87.66:38229), 4d7501ee179c45e5bfcf4c199d764f2e (127.1.87.65:39873)
I20260619 13:03:58.513739 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.508461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:54643 (local address 127.1.87.66:38229)
0619 13:03:58.508656 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.508660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.508677 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:03:58.508735 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.508738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.508793 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.508903 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:58.509356 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.510042 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.511986 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.512767 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.512918 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.513019 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.513331 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.513465 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:03:58.513748 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.507963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:55827 (local address 127.1.87.65:39873)
0619 13:03:58.508168 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.508175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.508367 (+ 192us) server_negotiation.cc:407] Connection header received
0619 13:03:58.508577 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.508581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.508638 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.508746 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:03:58.509260 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.510003 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.511945 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.512730 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.512892 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.513000 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.513332 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.513469 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:03:58.513783 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.507584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:38229 (local address 127.1.87.67:54643)
0619 13:03:58.508006 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:58.508030 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:58.508179 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:58.508923 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:58.508927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:58.508959 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:58.509191 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.509199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.510173 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.510179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.511829 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.511842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.512938 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.512944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.513171 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.513196 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 13:03:58.513304 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:03:58.513467 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":54}
I20260619 13:03:58.513918 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.507584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.1.87.67:55827)
0619 13:03:58.508268 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 13:03:58.508282 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:03:58.508429 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:03:58.508752 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:03:58.508757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:03:58.508782 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:03:58.509098 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.509108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.510150 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.510155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.511790 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:03:58.511802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.512938 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.512944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:03:58.513178 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.513203 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 13:03:58.513304 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 13:03:58.513470 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":213,"thread_start_us":255,"threads_started":1}
I20260619 13:03:58.514997 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f99d6ffda40e2a8bb743de57f029a" candidate_uuid: "b3464efc4ba74b31b68c23dd72959ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" is_pre_election: true
I20260619 13:03:58.515564 17038 raft_consensus.cc:2454] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3464efc4ba74b31b68c23dd72959ad9 in term 0.
I20260619 13:03:58.515177 17172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f99d6ffda40e2a8bb743de57f029a" candidate_uuid: "b3464efc4ba74b31b68c23dd72959ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b089d29fda264c4a88cdb082163ac8d4" is_pre_election: true
I20260619 13:03:58.515980 17172 raft_consensus.cc:2454] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3464efc4ba74b31b68c23dd72959ad9 in term 0.
I20260619 13:03:58.516317 17241 leader_election.cc:302] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d7501ee179c45e5bfcf4c199d764f2e, b3464efc4ba74b31b68c23dd72959ad9; no voters:
I20260619 13:03:58.516738 17375 raft_consensus.cc:2781] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:03:58.516851 17375 raft_consensus.cc:482] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:03:58.516916 17375 raft_consensus.cc:3037] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:58.519047 17375 raft_consensus.cc:504] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:03:58.519765 17375 leader_election.cc:288] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [CANDIDATE]: Term 1 election: Requested vote from peers b089d29fda264c4a88cdb082163ac8d4 (127.1.87.66:38229), 4d7501ee179c45e5bfcf4c199d764f2e (127.1.87.65:39873)
I20260619 13:03:58.520255 17172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f99d6ffda40e2a8bb743de57f029a" candidate_uuid: "b3464efc4ba74b31b68c23dd72959ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b089d29fda264c4a88cdb082163ac8d4"
I20260619 13:03:58.520366 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f99d6ffda40e2a8bb743de57f029a" candidate_uuid: "b3464efc4ba74b31b68c23dd72959ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7501ee179c45e5bfcf4c199d764f2e"
I20260619 13:03:58.520502 17172 raft_consensus.cc:3037] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:58.520587 17038 raft_consensus.cc:3037] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:03:58.523984 17172 raft_consensus.cc:2454] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3464efc4ba74b31b68c23dd72959ad9 in term 1.
I20260619 13:03:58.523984 17038 raft_consensus.cc:2454] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3464efc4ba74b31b68c23dd72959ad9 in term 1.
I20260619 13:03:58.524474 17241 leader_election.cc:302] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4d7501ee179c45e5bfcf4c199d764f2e, b089d29fda264c4a88cdb082163ac8d4, b3464efc4ba74b31b68c23dd72959ad9; no voters:
I20260619 13:03:58.524982 17375 raft_consensus.cc:2781] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:03:58.525638 17375 raft_consensus.cc:686] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 1 LEADER]: Becoming Leader. State: Replica: b3464efc4ba74b31b68c23dd72959ad9, State: Running, Role: LEADER
I20260619 13:03:58.526314 17375 consensus_queue.cc:237] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:03:58.531270 16894 catalog_manager.cc:5495] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3464efc4ba74b31b68c23dd72959ad9 (127.1.87.67). New cstate: current_term: 1 leader_uuid: "b3464efc4ba74b31b68c23dd72959ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } health_report { overall_health: UNKNOWN } } }
I20260619 13:03:58.564411 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.558426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.1.87.67:36423)
0619 13:03:58.558652 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.558658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.558680 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:03:58.558834 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.558838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.558916 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.559073 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:03:58.559525 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.560252 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.561744 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.562144 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.562282 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.562913 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:03:58.563727 (+ 814us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:03:58.563859 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.563983 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.564072 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:03:58.572865 17172 raft_consensus.cc:1261] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [term 1 FOLLOWER]: Refusing update from remote peer b3464efc4ba74b31b68c23dd72959ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:58.572865 17038 raft_consensus.cc:1261] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [term 1 FOLLOWER]: Refusing update from remote peer b3464efc4ba74b31b68c23dd72959ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:03:58.573944 17375 consensus_queue.cc:1035] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:03:58.574244 17379 consensus_queue.cc:1035] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:03:58.600953 16894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:03:58.601253 16894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:03:58.605664 16894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Sending DeleteTablet for 3 replicas of tablet ce1f99d6ffda40e2a8bb743de57f029a
I20260619 13:03:58.606920 17285 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1f99d6ffda40e2a8bb743de57f029a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:58 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:45038
I20260619 13:03:58.608335 17392 tablet_replica.cc:330] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: stopping tablet replica
I20260619 13:03:58.608260 17018 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1f99d6ffda40e2a8bb743de57f029a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:58 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:59538
I20260619 13:03:58.608430 17017 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1f99d6ffda40e2a8bb743de57f029a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:58 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:59538
I20260619 13:03:58.608834 17392 raft_consensus.cc:2229] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:03:58.608956 17152 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1f99d6ffda40e2a8bb743de57f029a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:03:58 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:60848
I20260619 13:03:58.609114 17393 tablet_replica.cc:330] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: stopping tablet replica
I20260619 13:03:58.609508 17392 raft_consensus.cc:2258] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:58.609483 17393 raft_consensus.cc:2229] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:03:58.609786 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 16955
I20260619 13:03:58.609877 17393 raft_consensus.cc:2258] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:58.611295 17394 tablet_replica.cc:330] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: stopping tablet replica
I20260619 13:03:58.611763 17394 raft_consensus.cc:2229] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:03:58.612287 17394 raft_consensus.cc:2258] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:03:58.613519 17392 ts_tablet_manager.cc:1905] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:58.614537 17393 ts_tablet_manager.cc:1905] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:58.622444 17392 ts_tablet_manager.cc:1918] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:58.622540 17393 ts_tablet_manager.cc:1918] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:58.622738 17393 log.cc:1196] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/ce1f99d6ffda40e2a8bb743de57f029a
I20260619 13:03:58.623282 17393 ts_tablet_manager.cc:1939] T ce1f99d6ffda40e2a8bb743de57f029a P 4d7501ee179c45e5bfcf4c199d764f2e: Deleting consensus metadata
I20260619 13:03:58.623646 17394 ts_tablet_manager.cc:1905] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:03:58.625301 16881 catalog_manager.cc:4841] TS 4d7501ee179c45e5bfcf4c199d764f2e (127.1.87.65:39873): tablet ce1f99d6ffda40e2a8bb743de57f029a (table test-table [id=1757acb7fb574f91b0cd214e4fefa7c3]) successfully deleted
Jun 19 13:03:58 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
W20260619 13:03:58.625910 16881 catalog_manager.cc:4808] TS 4d7501ee179c45e5bfcf4c199d764f2e (127.1.87.65:39873): delete failed for tablet ce1f99d6ffda40e2a8bb743de57f029a because the tablet was not found. No further retry: Not found: Tablet not found: ce1f99d6ffda40e2a8bb743de57f029a
I20260619 13:03:58.629395 17392 log.cc:1196] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/ce1f99d6ffda40e2a8bb743de57f029a
I20260619 13:03:58.630062 17392 ts_tablet_manager.cc:1939] T ce1f99d6ffda40e2a8bb743de57f029a P b3464efc4ba74b31b68c23dd72959ad9: Deleting consensus metadata
I20260619 13:03:58.630460 17394 ts_tablet_manager.cc:1918] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:03:58.630677 17394 log.cc:1196] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/ce1f99d6ffda40e2a8bb743de57f029a
I20260619 13:03:58.631243 17394 ts_tablet_manager.cc:1939] T ce1f99d6ffda40e2a8bb743de57f029a P b089d29fda264c4a88cdb082163ac8d4: Deleting consensus metadata
I20260619 13:03:58.632184 16881 catalog_manager.cc:4841] TS b3464efc4ba74b31b68c23dd72959ad9 (127.1.87.67:36423): tablet ce1f99d6ffda40e2a8bb743de57f029a (table test-table [id=1757acb7fb574f91b0cd214e4fefa7c3]) successfully deleted
I20260619 13:03:58.633106 16880 catalog_manager.cc:4841] TS b089d29fda264c4a88cdb082163ac8d4 (127.1.87.66:38229): tablet ce1f99d6ffda40e2a8bb743de57f029a (table test-table [id=1757acb7fb574f91b0cd214e4fefa7c3]) successfully deleted
I20260619 13:03:58.635403 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.611025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.1.87.65:39873)
0619 13:03:58.611206 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.611213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.611856 (+ 643us) server_negotiation.cc:407] Connection header received
0619 13:03:58.612167 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.612176 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.612269 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.612350 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:58.615546 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.616993 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.619976 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.620804 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.620910 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.627051 (+ 6141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:58.627065 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:58.627083 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:58.627136 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:58.631055 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.632290 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.632300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.632303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.632473 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.633378 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.633382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.633384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.633749 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:58.634298 (+ 549us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.634847 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.635032 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:03:58.636129 17078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:58.764746 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 16955
I20260619 13:03:58.782542 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 17089
Jun 19 13:03:58 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:03:58.801111 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.783404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.1.87.66:38229)
0619 13:03:58.783594 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.783601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.783624 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:03:58.783733 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.783739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.783828 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.783940 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:58.785517 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.786641 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.788009 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.788642 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.788770 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.793542 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:58.793562 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:58.793584 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:58.793637 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:58.797893 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.798758 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.798767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.798770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.798930 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.799291 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.799296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.799300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.799824 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:58.800114 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.800505 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.800781 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 13:03:58.801833 17212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:58.895133 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 17089
I20260619 13:03:58.919199 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 17222
Jun 19 13:03:58 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:03:58.935632 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:03:58.920162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.1.87.67:36423)
0619 13:03:58.920386 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 13:03:58.920391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:03:58.920405 (+ 14us) server_negotiation.cc:407] Connection header received
0619 13:03:58.920461 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:03:58.920465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:03:58.920526 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:03:58.920636 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:03:58.921885 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.922696 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.924168 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:03:58.924610 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:03:58.924748 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:03:58.929405 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:03:58.929417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:03:58.929432 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:03:58.929471 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:03:58.932776 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.933487 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.933498 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.933505 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.933665 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:03:58.934039 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:03:58.934044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:03:58.934049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:03:58.934413 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:03:58.934721 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 13:03:58.935020 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 13:03:58.935250 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:03:58.936561 17345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:59.061869 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 17222
I20260619 13:03:59.082109 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 16862
I20260619 13:03:59.083407 16924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:03:59.195130 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 16862
I20260619 13:04:19.217502 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:33205
--webserver_interface=127.1.87.126
--webserver_port=44323
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:04:19.570694 17400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:04:19.571098 17400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:04:19.571210 17400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:04:19.579983 17400 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 13:04:19.580119 17400 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 13:04:19.580161 17400 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 13:04:19.580207 17400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:04:19.580257 17400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:04:19.580330 17400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:04:19.580379 17400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:04:19.580456 17400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:04:19.592976 17400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:33205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=44323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:04:19.594417 17400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:04:19.595902 17400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:04:19.621590 17407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:19.621631 17409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:19.621770 17406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:04:19.623021 17400 server_base.cc:1034] running on GCE node
I20260619 13:04:19.624287 17400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:04:19.626047 17400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:04:19.627408 17400 hybrid_clock.cc:648] HybridClock initialized: now 1781874259627254 us; error 44 us; skew 500 ppm
I20260619 13:04:19.627852 17400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:04:19 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874259, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:04:19.635273 17400 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:04:19.637172 17400 webserver.cc:460] Webserver started at http://127.1.87.126:44323/ using document root <none> and password file <none>
I20260619 13:04:19.637988 17400 fs_manager.cc:359] Metadata directory not provided
I20260619 13:04:19.638131 17400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:04:19.645041 17400 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:04:19.648860 17416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:19.650395 17400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:04:19.650601 17400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "c4450e69c2a340db8416618432ad8b82"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "d9f34208c98f134be88562eb495e659e"
server_key_iv: "9d7e0a4c09877be253f3058c7b7d3d3e"
server_key_version: "encryptionkey@0"
I20260619 13:04:19.651839 17400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:04:19.678964 17400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:04:19.683647 17400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:04:19.683975 17400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:04:19.705160 17467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:33205 every 8 connection(s)
I20260619 13:04:19.705279 17400 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:33205
I20260619 13:04:19.707157 17400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:04:19.713513 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 17400
I20260619 13:04:19.714498 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:39873
--local_ip_for_outbound_sockets=127.1.87.65
--tserver_master_addrs=127.1.87.126:33205
--webserver_port=37031
--webserver_interface=127.1.87.65
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:04:19.715901 17468 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:04:19.721298 17468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Bootstrap starting.
I20260619 13:04:19.741640 17468 log.cc:824] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Log is configured to *not* fsync() on all Append() calls
I20260619 13:04:19.757546 17468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:04:19.758145 17468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Bootstrap complete.
I20260619 13:04:19.769820 17468 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:04:19.770707 17468 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4450e69c2a340db8416618432ad8b82, State: Initialized, Role: FOLLOWER
I20260619 13:04:19.771495 17468 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:04:19.771746 17468 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:04:19.771864 17468 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:04:19.772051 17468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:04:19.776170 17468 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:04:19.776752 17468 leader_election.cc:302] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4450e69c2a340db8416618432ad8b82; no voters:
I20260619 13:04:19.777401 17468 leader_election.cc:288] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:04:19.777540 17472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:04:19.778714 17472 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 2 LEADER]: Becoming Leader. State: Replica: c4450e69c2a340db8416618432ad8b82, State: Running, Role: LEADER
I20260619 13:04:19.779361 17472 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:04:19.779798 17468 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:04:19.781991 17474 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4450e69c2a340db8416618432ad8b82. Latest consensus state: current_term: 2 leader_uuid: "c4450e69c2a340db8416618432ad8b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } } }
I20260619 13:04:19.782461 17474 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: This master's current role is: LEADER
I20260619 13:04:19.783154 17473 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c4450e69c2a340db8416618432ad8b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } } }
I20260619 13:04:19.783571 17473 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: This master's current role is: LEADER
I20260619 13:04:19.796706 17488 catalog_manager.cc:1259] Loaded cluster ID: b980185d8bcc4567900d39aa9948155a
I20260619 13:04:19.796833 17488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: loading cluster ID for follower catalog manager: success
I20260619 13:04:19.801088 17488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: acquiring CA information for follower catalog manager: success
I20260619 13:04:19.804113 17488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:04:19.804421 17478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:04:19.810971 17478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:04:19.812244 17478 catalog_manager.cc:1259] Loaded cluster ID: b980185d8bcc4567900d39aa9948155a
I20260619 13:04:19.812340 17478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:04:19.814226 17478 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:04:19.815637 17478 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Loaded TSK: 0
I20260619 13:04:19.852123 17478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Generated new TSK 1
I20260619 13:04:19.852752 17478 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:04:20.104312 17470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:04:20.104593 17470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:04:20.104732 17470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:04:20.112201 17470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:04:20.112314 17470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:04:20.112516 17470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:04:20.122671 17470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:39873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=37031
--tserver_master_addrs=127.1.87.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:04:20.124034 17470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:04:20.125320 17470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:04:20.138175 17496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:20.138672 17495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:20.138986 17498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:04:20.139490 17470 server_base.cc:1034] running on GCE node
I20260619 13:04:20.140173 17470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:04:20.141279 17470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:04:20.142565 17470 hybrid_clock.cc:648] HybridClock initialized: now 1781874260142484 us; error 57 us; skew 500 ppm
I20260619 13:04:20.142949 17470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:04:20 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874260, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:04:20.146226 17470 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:04:20.148125 17470 webserver.cc:460] Webserver started at http://127.1.87.65:37031/ using document root <none> and password file <none>
I20260619 13:04:20.148826 17470 fs_manager.cc:359] Metadata directory not provided
I20260619 13:04:20.148980 17470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:04:20.154830 17470 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:04:20.158377 17505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:20.160367 17470 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:04:20.160619 17470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "4d7501ee179c45e5bfcf4c199d764f2e"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "841f18d81e351a86e26227b84a988c53"
server_key_iv: "47774dd07afdb8a983dc96ad1deb89b5"
server_key_version: "encryptionkey@0"
I20260619 13:04:20.161661 17470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:04:20.173760 17470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:04:20.178238 17470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:04:20.178547 17470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:04:20.179713 17470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:04:20.181169 17470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:04:20.181300 17470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:20.181397 17470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:04:20.181448 17470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:20.217163 17470 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:39873
I20260619 13:04:20.217279 17617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:39873 every 8 connection(s)
I20260619 13:04:20.218660 17470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:04:20.224741 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 17470
I20260619 13:04:20.225371 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:38229
--local_ip_for_outbound_sockets=127.1.87.66
--tserver_master_addrs=127.1.87.126:33205
--webserver_port=35915
--webserver_interface=127.1.87.66
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:04:20 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874260, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:04:20.245368 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:20.222068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:54257)
0619 13:04:20.223028 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:20.223071 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:04:20.225217 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:04:20.226637 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:04:20.226648 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:04:20.227261 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:04:20.227851 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:20.227867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:20.229554 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:20.229559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:20.230780 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:20.230792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:20.231870 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:20.231877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:20.232140 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:20.232864 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:04:20.232886 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:04:20.235275 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:04:20.240381 (+ 5106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:04:20.240389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:04:20.240403 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:04:20.240908 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:04:20.241565 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:04:20.241571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:04:20.241574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:04:20.242063 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:04:20.243206 (+ 1143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:04:20.243232 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:04:20.243662 (+ 430us) client_negotiation.cc:769] Sending connection context
0619 13:04:20.243924 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 13:04:20.244357 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":452,"threads_started":1}
I20260619 13:04:20.245978 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:20.221561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:54257 (local address 127.1.87.126:33205)
0619 13:04:20.222160 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:20.222173 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:20.223214 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 13:04:20.225785 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:20.225798 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:20.226121 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:20.226491 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:20.228055 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:20.229362 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:20.230925 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:20.231616 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:20.231823 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:20.235635 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:20.235648 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:20.235670 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:20.235729 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:20.240162 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:20.241084 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:20.241097 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:20.241107 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:20.241324 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:20.242217 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:20.242225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:20.242231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:20.242984 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:20.243208 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:20.244224 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0619 13:04:20.244717 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":284,"threads_started":1}
I20260619 13:04:20.249034 17618 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33205
I20260619 13:04:20.249374 17618 heartbeater.cc:464] Registering TS with master...
I20260619 13:04:20.250090 17618 heartbeater.cc:511] Master 127.1.87.126:33205 requested a full tablet report, sending...
I20260619 13:04:20.252702 17433 ts_manager.cc:194] Registered new tserver with Master: 4d7501ee179c45e5bfcf4c199d764f2e (127.1.87.65:39873)
I20260619 13:04:20.254851 17433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:54257
W20260619 13:04:20.573657 17622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:04:20.574010 17622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:04:20.574184 17622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:04:20.581787 17622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:04:20.581949 17622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:04:20.582139 17622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:04:20.592129 17622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:38229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=35915
--tserver_master_addrs=127.1.87.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:04:20.593458 17622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:04:20.594787 17622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:04:20.607861 17627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:20.607815 17628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:20.607831 17630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:04:20.608781 17622 server_base.cc:1034] running on GCE node
I20260619 13:04:20.609483 17622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:04:20.610561 17622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:04:20.611810 17622 hybrid_clock.cc:648] HybridClock initialized: now 1781874260611739 us; error 58 us; skew 500 ppm
I20260619 13:04:20.612150 17622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:04:20 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874260, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:04:20.615376 17622 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:04:20.616879 17622 webserver.cc:460] Webserver started at http://127.1.87.66:35915/ using document root <none> and password file <none>
I20260619 13:04:20.617589 17622 fs_manager.cc:359] Metadata directory not provided
I20260619 13:04:20.617702 17622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:04:20.623629 17622 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:04:20.626718 17637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:20.628310 17622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:04:20.628502 17622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "b089d29fda264c4a88cdb082163ac8d4"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "46e5eb33e158988733c7e28dd2020ce5"
server_key_iv: "d8e986b603b1ff583b4244d03495d851"
server_key_version: "encryptionkey@0"
I20260619 13:04:20.629514 17622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:04:20.660786 17622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:04:20.665236 17622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:04:20.665660 17622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:04:20.666736 17622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:04:20.668097 17622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:04:20.668182 17622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:20.668253 17622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:04:20.668308 17622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:20.704610 17622 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:38229
I20260619 13:04:20.704718 17749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:38229 every 8 connection(s)
I20260619 13:04:20.706274 17622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:04:20.713572 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 17622
I20260619 13:04:20.714351 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:36423
--local_ip_for_outbound_sockets=127.1.87.67
--tserver_master_addrs=127.1.87.126:33205
--webserver_port=46451
--webserver_interface=127.1.87.67
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:04:20 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874260, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:04:20.730448 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:20.709014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:37475 (local address 127.1.87.126:33205)
0619 13:04:20.709188 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:20.709193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:20.710471 (+ 1278us) server_negotiation.cc:407] Connection header received
0619 13:04:20.712488 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:20.712494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:20.712608 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:20.712719 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:20.714876 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:20.716006 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:20.718467 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:20.719183 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:20.719303 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:20.723518 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:20.723528 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:20.723535 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:20.723579 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:20.727087 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:20.727932 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:20.727940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:20.727946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:20.728092 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:20.728509 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:20.728517 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:20.728523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:20.728921 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:20.729074 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:20.729746 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 13:04:20.730005 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:04:20.731101 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:20.709537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:37475)
0619 13:04:20.710298 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:20.710340 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:04:20.712216 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:04:20.712950 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:04:20.712957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:04:20.713709 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:04:20.714673 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:20.714704 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:20.716188 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:20.716197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:20.718288 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:20.718305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:20.719449 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:20.719459 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:20.719788 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:20.720601 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:04:20.720634 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:04:20.723240 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:04:20.727237 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:04:20.727245 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:04:20.727258 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:04:20.727805 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:04:20.728201 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:04:20.728205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:04:20.728207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:04:20.728387 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:04:20.729044 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:04:20.729050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:04:20.729362 (+ 312us) client_negotiation.cc:769] Sending connection context
0619 13:04:20.729585 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 13:04:20.729969 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":330,"threads_started":1}
I20260619 13:04:20.732681 17750 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33205
I20260619 13:04:20.733009 17750 heartbeater.cc:464] Registering TS with master...
I20260619 13:04:20.733757 17750 heartbeater.cc:511] Master 127.1.87.126:33205 requested a full tablet report, sending...
I20260619 13:04:20.735610 17433 ts_manager.cc:194] Registered new tserver with Master: b089d29fda264c4a88cdb082163ac8d4 (127.1.87.66:38229)
I20260619 13:04:20.736459 17433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:37475
W20260619 13:04:21.042615 17753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:04:21.042929 17753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:04:21.043043 17753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:04:21.050289 17753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:04:21.050469 17753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:04:21.050599 17753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:04:21.060467 17753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:36423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=46451
--tserver_master_addrs=127.1.87.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:04:21.061802 17753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:04:21.063055 17753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:04:21.076001 17758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:21.076074 17761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:21.076031 17759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:04:21.077096 17753 server_base.cc:1034] running on GCE node
I20260619 13:04:21.077711 17753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:04:21.078711 17753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:04:21.079964 17753 hybrid_clock.cc:648] HybridClock initialized: now 1781874261079921 us; error 39 us; skew 500 ppm
I20260619 13:04:21.080308 17753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:04:21 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874261, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:04:21.083463 17753 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:04:21.084967 17753 webserver.cc:460] Webserver started at http://127.1.87.67:46451/ using document root <none> and password file <none>
I20260619 13:04:21.085641 17753 fs_manager.cc:359] Metadata directory not provided
I20260619 13:04:21.085804 17753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:04:21.091228 17753 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:04:21.094143 17768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:21.095597 17753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:04:21.095777 17753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "b3464efc4ba74b31b68c23dd72959ad9"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "b46ad94f393166f7749877c23d1f7e8c"
server_key_iv: "1e3e42f28cff1afa5e605ec77798b9bd"
server_key_version: "encryptionkey@0"
I20260619 13:04:21.096714 17753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:04:21.114437 17753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:04:21.118518 17753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:04:21.118796 17753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:04:21.119808 17753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:04:21.121068 17753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:04:21.121196 17753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:21.121343 17753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:04:21.121431 17753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:21.155184 17753 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:36423
I20260619 13:04:21.155273 17880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:36423 every 8 connection(s)
I20260619 13:04:21.156697 17753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:04:21.162860 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 17753
Jun 19 13:04:21 dist-test-slave-tpsm krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874261, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:04:21.178292 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.164010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.1.87.126:33205)
0619 13:04:21.164747 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.164754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.164775 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:04:21.164873 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.164877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.164952 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.165070 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:21.166617 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.167419 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.168823 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.169513 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.169642 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.172416 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:21.172422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:21.172428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:21.172471 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:21.175636 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.176279 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.176285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.176288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.176401 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.176779 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.176785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.176788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.177037 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:21.177306 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.177602 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.177836 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"spinlock_wait_cycles":3840,"thread_start_us":502,"threads_started":1}
I20260619 13:04:21.178398 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.159293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:47219 (local address 127.1.87.126:33205)
0619 13:04:21.159535 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.159542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.160777 (+ 1235us) server_negotiation.cc:407] Connection header received
0619 13:04:21.162785 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.162789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.162855 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.162932 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:21.165118 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.165850 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.167477 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.167873 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.168041 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.171906 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:21.171911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:21.171914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:21.171944 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:21.175078 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.175920 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.175927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.175935 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.176043 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.176486 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.176491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.176494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.176751 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:21.176912 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.177620 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.178015 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":103,"spinlock_wait_cycles":2816}
I20260619 13:04:21.179459 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.159856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:47219)
0619 13:04:21.160595 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.160639 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:04:21.162449 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:04:21.163218 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:04:21.163229 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:04:21.164111 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:04:21.164936 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.164969 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.166013 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.166020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.167255 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.167267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.168098 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.168107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.168370 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.169073 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:04:21.169104 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:04:21.171693 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:04:21.175255 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:04:21.175265 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:04:21.175282 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:04:21.175768 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:04:21.176169 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:04:21.176174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:04:21.176178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:04:21.176364 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:04:21.176886 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:04:21.176894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:04:21.177237 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 13:04:21.177500 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 13:04:21.178021 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":274,"threads_started":1}
I20260619 13:04:21.180953 17881 heartbeater.cc:347] Connected to a master server at 127.1.87.126:33205
I20260619 13:04:21.181211 17881 heartbeater.cc:464] Registering TS with master...
I20260619 13:04:21.181855 17881 heartbeater.cc:511] Master 127.1.87.126:33205 requested a full tablet report, sending...
I20260619 13:04:21.183175 17433 ts_manager.cc:194] Registered new tserver with Master: b3464efc4ba74b31b68c23dd72959ad9 (127.1.87.67:36423)
I20260619 13:04:21.183983 17433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:47219
I20260619 13:04:21.193301 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:04:21.199801 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.194294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.1.87.126:33205)
0619 13:04:21.194574 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.194578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.194901 (+ 323us) server_negotiation.cc:407] Connection header received
0619 13:04:21.195086 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.195090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.195242 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.195447 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:04:21.195900 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.196644 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.198093 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.198499 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.198591 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.199038 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:04:21.199236 (+ 198us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 13:04:21.199601 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51238: token expired
Metrics: {"server-negotiator.queue_time_us":162}
W20260619 13:04:21.199941 17620 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51238: token expired
W20260619 13:04:21.200450 17357 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.87.126:33205)
I20260619 13:04:21.214283 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.200933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.1.87.126:33205)
0619 13:04:21.201096 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.201102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.201181 (+ 79us) server_negotiation.cc:407] Connection header received
0619 13:04:21.201361 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.201366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.201435 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.201559 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:21.202930 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.203734 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.205131 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.205557 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.205692 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.208487 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:21.208500 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:21.208503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:21.208534 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:21.211016 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.211792 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.211798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.211801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.211931 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.212315 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.212321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.212324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.212827 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:21.213134 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.213694 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.213931 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:04:21.218180 17433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:04:21.226599 17433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:04:21.259373 17618 heartbeater.cc:503] Master 127.1.87.126:33205 was elected leader, sending a full tablet report...
I20260619 13:04:21.264071 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.239869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.1.87.66:38229)
0619 13:04:21.256062 (+ 16193us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.256070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.256091 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:04:21.256188 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.256195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.256451 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.256651 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:04:21.257593 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.258755 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.261871 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.262831 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.263005 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.263216 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.263554 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.263743 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16041,"thread_start_us":16055,"threads_started":1}
I20260619 13:04:21.264645 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.239714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:38229 (local address 127.0.0.1:46606)
0619 13:04:21.251182 (+ 11468us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.251262 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 13:04:21.251570 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:04:21.256827 (+ 5257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:04:21.256834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:04:21.256863 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:04:21.257364 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.257380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.258888 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.258897 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.261724 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.261736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.263067 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.263074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.263279 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.263360 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 13:04:21.263514 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 13:04:21.263737 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11057,"thread_start_us":12116,"threads_started":1}
I20260619 13:04:21.267385 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.253248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.1.87.65:39873)
0619 13:04:21.256304 (+ 3056us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.256313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.256336 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:04:21.256485 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.256493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.256726 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.256905 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:04:21.257978 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.259539 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.264250 (+ 4711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.265761 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.265890 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.266113 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.266906 (+ 793us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.267074 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2896,"thread_start_us":1929,"threads_started":1}
I20260619 13:04:21.267735 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.239831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36423 (local address 127.0.0.1:55960)
0619 13:04:21.253372 (+ 13541us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.253397 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:04:21.253919 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:04:21.256989 (+ 3070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:04:21.256996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:04:21.257026 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:04:21.257552 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.257564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.259632 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.259639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.264054 (+ 4415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.264068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.265985 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.265993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.266216 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.266298 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 13:04:21.266570 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 13:04:21.266758 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12571,"mutex_wait_us":1883,"spinlock_wait_cycles":2176,"thread_start_us":12686,"threads_started":1}
I20260619 13:04:21.267760 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.253046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:56162)
0619 13:04:21.255720 (+ 2674us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.255735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:04:21.255856 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:04:21.257136 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:04:21.257139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:04:21.257173 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:04:21.257796 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.257811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.259698 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.259703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.264054 (+ 4351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.264068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.266226 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.266232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.266640 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.266680 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 13:04:21.266823 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 13:04:21.267088 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2566,"mutex_wait_us":1967,"spinlock_wait_cycles":896,"thread_start_us":2401,"threads_started":1}
I20260619 13:04:21.268885 17685 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8981e959b941638f69079fb2fe2fc5 (DEFAULT_TABLE table=test-table [id=d71d81b0c4ee4737832d155086b3c2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:04:21.271191 17685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8981e959b941638f69079fb2fe2fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:04:21.271256 17553 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8981e959b941638f69079fb2fe2fc5 (DEFAULT_TABLE table=test-table [id=d71d81b0c4ee4737832d155086b3c2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:04:21.272028 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.240468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.1.87.67:36423)
0619 13:04:21.256030 (+ 15562us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.256042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.256080 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:04:21.256182 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.256190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.256452 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.256651 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:04:21.257755 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.259454 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.264266 (+ 4812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.265761 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.265890 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.271572 (+ 5682us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.271636 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.271777 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15142,"thread_start_us":15375,"threads_started":1}
I20260619 13:04:21.272619 17553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8981e959b941638f69079fb2fe2fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:04:21.274777 17816 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8981e959b941638f69079fb2fe2fc5 (DEFAULT_TABLE table=test-table [id=d71d81b0c4ee4737832d155086b3c2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:04:21.276149 17816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8981e959b941638f69079fb2fe2fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:04:21.287200 17897 tablet_bootstrap.cc:492] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Bootstrap starting.
I20260619 13:04:21.288082 17898 tablet_bootstrap.cc:492] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: Bootstrap starting.
I20260619 13:04:21.289389 17899 tablet_bootstrap.cc:492] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: Bootstrap starting.
I20260619 13:04:21.292387 17899 tablet_bootstrap.cc:654] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: Neither blocks nor log segments found. Creating new log.
I20260619 13:04:21.292651 17897 tablet_bootstrap.cc:654] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Neither blocks nor log segments found. Creating new log.
I20260619 13:04:21.293452 17899 log.cc:824] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:04:21.293562 17897 log.cc:824] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:04:21.295869 17898 tablet_bootstrap.cc:654] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: Neither blocks nor log segments found. Creating new log.
I20260619 13:04:21.296245 17899 tablet_bootstrap.cc:492] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: No bootstrap required, opened a new log
I20260619 13:04:21.296515 17899 ts_tablet_manager.cc:1397] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 13:04:21.296904 17898 log.cc:824] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: Log is configured to *not* fsync() on all Append() calls
I20260619 13:04:21.299669 17897 tablet_bootstrap.cc:492] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: No bootstrap required, opened a new log
I20260619 13:04:21.299666 17898 tablet_bootstrap.cc:492] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: No bootstrap required, opened a new log
I20260619 13:04:21.299934 17898 ts_tablet_manager.cc:1397] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260619 13:04:21.299934 17897 ts_tablet_manager.cc:1397] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260619 13:04:21.304692 17899 raft_consensus.cc:348] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:04:21.305083 17899 raft_consensus.cc:374] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:04:21.305222 17899 raft_consensus.cc:729] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3464efc4ba74b31b68c23dd72959ad9, State: Initialized, Role: FOLLOWER
I20260619 13:04:21.306234 17899 consensus_queue.cc:260] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:04:21.307451 17881 heartbeater.cc:503] Master 127.1.87.126:33205 was elected leader, sending a full tablet report...
I20260619 13:04:21.308159 17899 ts_tablet_manager.cc:1428] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 13:04:21.310842 17897 raft_consensus.cc:348] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:04:21.310842 17898 raft_consensus.cc:348] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:04:21.311303 17898 raft_consensus.cc:374] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:04:21.311304 17897 raft_consensus.cc:374] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:04:21.311435 17898 raft_consensus.cc:729] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b089d29fda264c4a88cdb082163ac8d4, State: Initialized, Role: FOLLOWER
I20260619 13:04:21.311435 17897 raft_consensus.cc:729] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7501ee179c45e5bfcf4c199d764f2e, State: Initialized, Role: FOLLOWER
I20260619 13:04:21.312094 17897 consensus_queue.cc:260] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:04:21.312094 17898 consensus_queue.cc:260] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:04:21.313923 17750 heartbeater.cc:503] Master 127.1.87.126:33205 was elected leader, sending a full tablet report...
I20260619 13:04:21.314065 17897 ts_tablet_manager.cc:1428] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 13:04:21.314276 17898 ts_tablet_manager.cc:1428] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260619 13:04:21.408351 17882 tablet.cc:2369] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:04:21.458447 17751 tablet.cc:2369] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:04:21.471087 17619 tablet.cc:2369] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:04:21.494998 17905 raft_consensus.cc:482] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:04:21.495353 17905 raft_consensus.cc:504] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:04:21.497012 17905 leader_election.cc:288] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3464efc4ba74b31b68c23dd72959ad9 (127.1.87.67:36423), 4d7501ee179c45e5bfcf4c199d764f2e (127.1.87.65:39873)
I20260619 13:04:21.503690 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.497536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:33201 (local address 127.1.87.67:36423)
0619 13:04:21.497729 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.497734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.498246 (+ 512us) server_negotiation.cc:407] Connection header received
0619 13:04:21.498482 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.498487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.498582 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.498712 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:04:21.499277 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.500054 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.501960 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.502740 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.502891 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.503003 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.503282 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.503436 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:04:21.503799 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.497309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:36423 (local address 127.1.87.66:33201)
0619 13:04:21.498141 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.498161 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:04:21.498317 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:04:21.498791 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:04:21.498795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:04:21.498825 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:04:21.499120 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.499129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.500191 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.500196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.501795 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.501813 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.502953 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.502959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.503125 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.503148 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:04:21.503255 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 13:04:21.503419 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":380,"threads_started":1}
I20260619 13:04:21.504170 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.498433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:55559 (local address 127.1.87.65:39873)
0619 13:04:21.498680 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.498687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.499049 (+ 362us) server_negotiation.cc:407] Connection header received
0619 13:04:21.499266 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.499269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.499374 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.499476 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:04:21.500007 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.500777 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.502472 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.503219 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.503399 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.503502 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.503807 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.503930 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:04:21.504227 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.498251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.1.87.66:55559)
0619 13:04:21.498942 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.498959 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:04:21.499114 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:04:21.499490 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:04:21.499495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:04:21.499516 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:04:21.499853 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.499861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.500898 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.500906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.502312 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:04:21.502324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.503455 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.503461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:04:21.503638 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.503662 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:04:21.503770 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:04:21.503891 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":387,"threads_started":1}
I20260619 13:04:21.505033 17836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8981e959b941638f69079fb2fe2fc5" candidate_uuid: "b089d29fda264c4a88cdb082163ac8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3464efc4ba74b31b68c23dd72959ad9" is_pre_election: true
I20260619 13:04:21.505308 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8981e959b941638f69079fb2fe2fc5" candidate_uuid: "b089d29fda264c4a88cdb082163ac8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" is_pre_election: true
I20260619 13:04:21.505640 17836 raft_consensus.cc:2454] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b089d29fda264c4a88cdb082163ac8d4 in term 0.
I20260619 13:04:21.505800 17573 raft_consensus.cc:2454] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b089d29fda264c4a88cdb082163ac8d4 in term 0.
I20260619 13:04:21.506326 17641 leader_election.cc:302] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b089d29fda264c4a88cdb082163ac8d4, b3464efc4ba74b31b68c23dd72959ad9; no voters:
I20260619 13:04:21.506651 17905 raft_consensus.cc:2781] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:04:21.506814 17905 raft_consensus.cc:482] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:04:21.506938 17905 raft_consensus.cc:3037] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:04:21.509066 17905 raft_consensus.cc:504] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:04:21.509697 17905 leader_election.cc:288] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [CANDIDATE]: Term 1 election: Requested vote from peers b3464efc4ba74b31b68c23dd72959ad9 (127.1.87.67:36423), 4d7501ee179c45e5bfcf4c199d764f2e (127.1.87.65:39873)
I20260619 13:04:21.510200 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8981e959b941638f69079fb2fe2fc5" candidate_uuid: "b089d29fda264c4a88cdb082163ac8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7501ee179c45e5bfcf4c199d764f2e"
I20260619 13:04:21.510222 17836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8981e959b941638f69079fb2fe2fc5" candidate_uuid: "b089d29fda264c4a88cdb082163ac8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3464efc4ba74b31b68c23dd72959ad9"
I20260619 13:04:21.510401 17573 raft_consensus.cc:3037] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:04:21.510412 17836 raft_consensus.cc:3037] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:04:21.512542 17836 raft_consensus.cc:2454] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b089d29fda264c4a88cdb082163ac8d4 in term 1.
I20260619 13:04:21.512606 17573 raft_consensus.cc:2454] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b089d29fda264c4a88cdb082163ac8d4 in term 1.
I20260619 13:04:21.512933 17641 leader_election.cc:302] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b089d29fda264c4a88cdb082163ac8d4, b3464efc4ba74b31b68c23dd72959ad9; no voters:
I20260619 13:04:21.513206 17905 raft_consensus.cc:2781] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:04:21.513638 17905 raft_consensus.cc:686] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 1 LEADER]: Becoming Leader. State: Replica: b089d29fda264c4a88cdb082163ac8d4, State: Running, Role: LEADER
I20260619 13:04:21.514124 17905 consensus_queue.cc:237] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } }
I20260619 13:04:21.518569 17433 catalog_manager.cc:5495] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b089d29fda264c4a88cdb082163ac8d4 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "b089d29fda264c4a88cdb082163ac8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b089d29fda264c4a88cdb082163ac8d4" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 } health_report { overall_health: UNKNOWN } } }
I20260619 13:04:21.560989 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.555714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.1.87.66:38229)
0619 13:04:21.555922 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.555930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.555955 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:04:21.556234 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.556238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.556313 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.556480 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:04:21.556893 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.557625 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.558932 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.559374 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.559471 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.559904 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:04:21.560398 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:04:21.560504 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.560596 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.560660 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:04:21.569135 17573 raft_consensus.cc:1261] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [term 1 FOLLOWER]: Refusing update from remote peer b089d29fda264c4a88cdb082163ac8d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:04:21.569222 17836 raft_consensus.cc:1261] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [term 1 FOLLOWER]: Refusing update from remote peer b089d29fda264c4a88cdb082163ac8d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:04:21.570147 17908 consensus_queue.cc:1035] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7501ee179c45e5bfcf4c199d764f2e" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:04:21.570525 17905 consensus_queue.cc:1035] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3464efc4ba74b31b68c23dd72959ad9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:04:21.582201 17911 mvcc.cc:205] Tried to move back new op lower bound from 7298556975377416192 to 7298556975165714432. Current Snapshot: MvccSnapshot[applied={T|T < 7298556975377416192}]
I20260619 13:04:21.596216 17432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:04:21.596473 17432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:04:21.600235 17432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Sending DeleteTablet for 3 replicas of tablet 8a8981e959b941638f69079fb2fe2fc5
I20260619 13:04:21.601480 17685 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8981e959b941638f69079fb2fe2fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:04:21 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:46606
I20260619 13:04:21.601692 17816 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8981e959b941638f69079fb2fe2fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:04:21 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:55960
I20260619 13:04:21.601778 17553 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8981e959b941638f69079fb2fe2fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:04:21 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:56162
I20260619 13:04:21.603130 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 17470
I20260619 13:04:21.612293 17920 tablet_replica.cc:330] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: stopping tablet replica
I20260619 13:04:21.613734 17920 raft_consensus.cc:2229] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:04:21.615536 17920 raft_consensus.cc:2258] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:04:21.617153 17922 tablet_replica.cc:330] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: stopping tablet replica
I20260619 13:04:21.617744 17922 raft_consensus.cc:2229] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:04:21.618294 17922 raft_consensus.cc:2258] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:04:21.619130 17921 tablet_replica.cc:330] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: stopping tablet replica
I20260619 13:04:21.619582 17921 raft_consensus.cc:2229] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:04:21.620100 17921 raft_consensus.cc:2258] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:04:21.620811 17922 ts_tablet_manager.cc:1905] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:04:21.621058 17920 ts_tablet_manager.cc:1905] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:04:21.625831 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.604142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.1.87.65:39873)
0619 13:04:21.604373 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.604383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.604824 (+ 441us) server_negotiation.cc:407] Connection header received
0619 13:04:21.604892 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.604897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.605005 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.605121 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:21.607112 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.608352 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.610511 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.611185 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.611346 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.615369 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:21.615389 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:21.615409 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:21.615457 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:21.621121 (+ 5664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.622118 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.622127 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.622131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.622322 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.622834 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.622839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.622843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.623380 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:21.624899 (+ 1519us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.625061 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.625325 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:04:21.627959 17921 ts_tablet_manager.cc:1905] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:04:21.628378 17920 ts_tablet_manager.cc:1918] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:04:21.628616 17920 log.cc:1196] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/8a8981e959b941638f69079fb2fe2fc5
I20260619 13:04:21.628844 17613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:04:21.628903 17922 ts_tablet_manager.cc:1918] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:04:21.629127 17922 log.cc:1196] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/8a8981e959b941638f69079fb2fe2fc5
I20260619 13:04:21.629209 17920 ts_tablet_manager.cc:1939] T 8a8981e959b941638f69079fb2fe2fc5 P b089d29fda264c4a88cdb082163ac8d4: Deleting consensus metadata
I20260619 13:04:21.631508 17419 catalog_manager.cc:4841] TS b089d29fda264c4a88cdb082163ac8d4 (127.1.87.66:38229): tablet 8a8981e959b941638f69079fb2fe2fc5 (table test-table [id=d71d81b0c4ee4737832d155086b3c2ab]) successfully deleted
I20260619 13:04:21.635526 17921 ts_tablet_manager.cc:1918] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:04:21.635710 17921 log.cc:1196] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/8a8981e959b941638f69079fb2fe2fc5
I20260619 13:04:21.636159 17921 ts_tablet_manager.cc:1939] T 8a8981e959b941638f69079fb2fe2fc5 P b3464efc4ba74b31b68c23dd72959ad9: Deleting consensus metadata
I20260619 13:04:21.637735 17420 catalog_manager.cc:4841] TS b3464efc4ba74b31b68c23dd72959ad9 (127.1.87.67:36423): tablet 8a8981e959b941638f69079fb2fe2fc5 (table test-table [id=d71d81b0c4ee4737832d155086b3c2ab]) successfully deleted
I20260619 13:04:21.770507 17922 ts_tablet_manager.cc:1939] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Deleting consensus metadata
I20260619 13:04:21.770619 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 17470
I20260619 13:04:21.790927 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 17622
W20260619 13:04:21.790880 17420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.87.65:39873 (error 108)
I20260619 13:04:21.793143 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.792352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:56170)
0619 13:04:21.792800 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.792924 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":222,"threads_started":1}
W20260619 13:04:21.793426 17420 catalog_manager.cc:4584] TS 4d7501ee179c45e5bfcf4c199d764f2e (127.1.87.65:39873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a8981e959b941638f69079fb2fe2fc5: Network error: Client connection negotiation failed: client connection to 127.1.87.65:39873: connect: Connection refused (error 111)
I20260619 13:04:21.807983 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.792008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.1.87.66:38229)
0619 13:04:21.792227 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.792245 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.792265 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:04:21.792341 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.792346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.792431 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.792582 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:21.794160 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.795458 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.796959 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.797606 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.797748 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.800733 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:21.800751 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:21.800771 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:21.800813 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:21.804927 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.805656 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.805667 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.805671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.805866 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.806271 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.806276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.806280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.806730 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:21.807062 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.807302 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.807555 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:04:21.808775 17745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:04:21.847378 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.846720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:56176)
0619 13:04:21.847012 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.847163 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260619 13:04:21.848526 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.848134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:56184)
0619 13:04:21.848287 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.848370 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 13:04:21.909328 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.908785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:56198)
0619 13:04:21.909046 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.909144 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 13:04:21.910629 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.910286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:56202)
0619 13:04:21.910425 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.910493 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 13:04:21.926033 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 17622
I20260619 13:04:21.941819 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 17753
I20260619 13:04:21.956823 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.942829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.1.87.67:36423)
0619 13:04:21.943013 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:21.943018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:21.943036 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:04:21.943091 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:21.943095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:21.943156 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:21.943268 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:21.944937 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.945800 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.947166 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:21.947657 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:21.947792 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:21.950404 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:21.950418 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:21.950435 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:21.950477 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:21.953475 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.954244 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.954252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.954254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.954549 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:21.955103 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:21.955110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:21.955114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:21.955560 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:21.955903 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:21.956232 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 13:04:21.956464 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:04:21.957580 17876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:04:21.991065 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.990592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:56218)
0619 13:04:21.990804 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.990886 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 13:04:21.992162 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:21.991776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:56234)
0619 13:04:21.991975 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:21.992038 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 13:04:22.050052 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 17753
I20260619 13:04:22.070278 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 17400
I20260619 13:04:22.071578 17463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:04:22.186012 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:22.185415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39873 (local address 127.0.0.1:56242)
0619 13:04:22.185656 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:22.185787 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.65:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 13:04:22.186010 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 17400
I20260619 13:04:42.210067 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:33205
--webserver_interface=127.1.87.126
--webserver_port=44323
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:04:42.555491 17929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:04:42.555768 17929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:04:42.555922 17929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:04:42.562959 17929 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 13:04:42.563045 17929 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 13:04:42.563119 17929 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 13:04:42.563201 17929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:04:42.563301 17929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:04:42.563429 17929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:04:42.563521 17929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:04:42.563601 17929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:04:42.573254 17929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:33205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=44323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:04:42.574595 17929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:04:42.575896 17929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:04:42.587656 17934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:42.587702 17937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:42.587600 17935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:04:42.588308 17929 server_base.cc:1034] running on GCE node
I20260619 13:04:42.589197 17929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:04:42.590603 17929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:04:42.591859 17929 hybrid_clock.cc:648] HybridClock initialized: now 1781874282591794 us; error 57 us; skew 500 ppm
I20260619 13:04:42.592242 17929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:04:42 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874282, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:04:42.599552 17929 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:04:42.600929 17929 webserver.cc:460] Webserver started at http://127.1.87.126:44323/ using document root <none> and password file <none>
I20260619 13:04:42.601572 17929 fs_manager.cc:359] Metadata directory not provided
I20260619 13:04:42.601655 17929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:04:42.607283 17929 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 13:04:42.610101 17944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:42.611461 17929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:04:42.611627 17929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "c4450e69c2a340db8416618432ad8b82"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "d9f34208c98f134be88562eb495e659e"
server_key_iv: "9d7e0a4c09877be253f3058c7b7d3d3e"
server_key_version: "encryptionkey@0"
I20260619 13:04:42.612551 17929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:04:42.623390 17929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:04:42.627622 17929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:04:42.627907 17929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:04:42.645809 17929 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:33205
I20260619 13:04:42.645844 17995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:33205 every 8 connection(s)
I20260619 13:04:42.647243 17929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:04:42.654039 17996 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:04:42.655416 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 17929
I20260619 13:04:42.656193 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:39873
--local_ip_for_outbound_sockets=127.1.87.65
--tserver_master_addrs=127.1.87.126:33205
--webserver_port=37031
--webserver_interface=127.1.87.65
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:04:42.657394 17996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Bootstrap starting.
I20260619 13:04:42.685014 17996 log.cc:824] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Log is configured to *not* fsync() on all Append() calls
I20260619 13:04:42.706455 17996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:04:42.707147 17996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Bootstrap complete.
I20260619 13:04:42.720729 17996 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:04:42.721666 17996 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4450e69c2a340db8416618432ad8b82, State: Initialized, Role: FOLLOWER
I20260619 13:04:42.722267 17996 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:04:42.722497 17996 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:04:42.722615 17996 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:04:42.722812 17996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:04:42.726681 17996 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:04:42.727173 17996 leader_election.cc:302] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4450e69c2a340db8416618432ad8b82; no voters:
I20260619 13:04:42.727886 17996 leader_election.cc:288] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 13:04:42.728027 18000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 13:04:42.730263 17996 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:04:42.731590 18000 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [term 3 LEADER]: Becoming Leader. State: Replica: c4450e69c2a340db8416618432ad8b82, State: Running, Role: LEADER
I20260619 13:04:42.732276 18000 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } }
I20260619 13:04:42.734930 18001 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c4450e69c2a340db8416618432ad8b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } } }
I20260619 13:04:42.735634 18001 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: This master's current role is: LEADER
I20260619 13:04:42.736127 18002 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4450e69c2a340db8416618432ad8b82. Latest consensus state: current_term: 3 leader_uuid: "c4450e69c2a340db8416618432ad8b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4450e69c2a340db8416618432ad8b82" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 33205 } } }
I20260619 13:04:42.736660 18002 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82 [sys.catalog]: This master's current role is: LEADER
I20260619 13:04:42.738037 18008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:04:42.748076 18008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:04:42.750147 18008 catalog_manager.cc:1259] Loaded cluster ID: b980185d8bcc4567900d39aa9948155a
I20260619 13:04:42.750329 18008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:04:42.756007 18008 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:04:42.758229 18008 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Loaded TSK: 0
I20260619 13:04:42.758510 18008 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c4450e69c2a340db8416618432ad8b82: Loaded TSK: 1
F20260619 13:04:42.760098 18008 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9bfd0ab9ad google::LogMessage::Fail() at ??:0
@ 0x7f9bfd0ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9bfd0ab4de google::LogMessage::Flush() at ??:0
@ 0x7f9bfd0aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c00e0f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9c00dfe73e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260619 13:04:42.795987 18019 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f9c12ada1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9c12ad9783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9c12b652a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9c12b650c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f9c12baf5e9 std::function<>::operator()() at ??:0
@ 0x7f9c12addcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9c12adbc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9c12b63610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9c12b63439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9bffcf7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9bffcfa91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9bffcfa799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9bffcd5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c02faf6db start_thread at ??:0
@ 0x7f9bfbf4561f clone at ??:0
W20260619 13:04:43.063825 17998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:04:43.064143 17998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:04:43.064311 17998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:04:43.071698 17998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:04:43.071847 17998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:04:43.072028 17998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:04:43.081708 17998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:39873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=37031
--tserver_master_addrs=127.1.87.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:04:43.083053 17998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:04:43.084372 17998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:04:43.097446 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:43.097498 18046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:43.097520 18044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:04:43.098783 17998 server_base.cc:1034] running on GCE node
I20260619 13:04:43.099550 17998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:04:43.100639 17998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:04:43.101904 17998 hybrid_clock.cc:648] HybridClock initialized: now 1781874283101833 us; error 59 us; skew 500 ppm
I20260619 13:04:43.102246 17998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:04:43 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:04:43.105436 17998 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:04:43.107029 17998 webserver.cc:460] Webserver started at http://127.1.87.65:37031/ using document root <none> and password file <none>
I20260619 13:04:43.107803 17998 fs_manager.cc:359] Metadata directory not provided
I20260619 13:04:43.107952 17998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:04:43.113425 17998 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:04:43.116255 18053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:43.117551 17998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:04:43.117741 17998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "4d7501ee179c45e5bfcf4c199d764f2e"
format_stamp: "Formatted at 2026-06-19 13:03:56 on dist-test-slave-tpsm"
server_key: "841f18d81e351a86e26227b84a988c53"
server_key_iv: "47774dd07afdb8a983dc96ad1deb89b5"
server_key_version: "encryptionkey@0"
I20260619 13:04:43.118733 17998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:04:43.162353 17998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:04:43.166880 17998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:04:43.167236 17998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:04:43.168290 17998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:04:43.170244 18060 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 13:04:43.175689 18060 ts_tablet_manager.cc:1852] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 13:04:43.176680 18060 ts_tablet_manager.cc:1905] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:04:43.183844 18060 ts_tablet_manager.cc:1918] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:04:43.184127 18060 ts_tablet_manager.cc:1939] T 8a8981e959b941638f69079fb2fe2fc5 P 4d7501ee179c45e5bfcf4c199d764f2e: Deleting consensus metadata
I20260619 13:04:43.184592 17998 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 13:04:43.184746 17998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.001s
I20260619 13:04:43.184892 17998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:04:43.185004 17998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:43.220584 17998 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:39873
I20260619 13:04:43.220688 18166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:39873 every 8 connection(s)
I20260619 13:04:43.222131 17998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:04:43.227473 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:43.225413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:46763)
0619 13:04:43.226183 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:43.226501 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":325,"threads_started":1}
I20260619 13:04:43.228129 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 17998
W20260619 13:04:43.228317 18167 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:33205 (0 consecutive failures): Network error: Failed to ping master at 127.1.87.126:33205: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
I20260619 13:04:43.228893 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:38229
--local_ip_for_outbound_sockets=127.1.87.66
--tserver_master_addrs=127.1.87.126:33205
--webserver_port=35915
--webserver_interface=127.1.87.66
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:04:43.229264 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:43.228843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:56793)
0619 13:04:43.229028 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:43.229128 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 13:04:43.230293 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:43.229861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:52839)
0619 13:04:43.230027 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:43.230127 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:04:43.230556 18167 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:04:43.581790 18170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:04:43.582072 18170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:04:43.582266 18170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:04:43.589520 18170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:04:43.589628 18170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:04:43.589830 18170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:04:43.600090 18170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:38229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=35915
--tserver_master_addrs=127.1.87.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:04:43.601629 18170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:04:43.603107 18170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:04:43.616293 18178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:43.616357 18175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:43.616382 18176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:04:43.617385 18170 server_base.cc:1034] running on GCE node
I20260619 13:04:43.617995 18170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:04:43.618983 18170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:04:43.620219 18170 hybrid_clock.cc:648] HybridClock initialized: now 1781874283620170 us; error 58 us; skew 500 ppm
I20260619 13:04:43.620555 18170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:04:43 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:04:43.623939 18170 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:04:43.625357 18170 webserver.cc:460] Webserver started at http://127.1.87.66:35915/ using document root <none> and password file <none>
I20260619 13:04:43.626081 18170 fs_manager.cc:359] Metadata directory not provided
I20260619 13:04:43.626235 18170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:04:43.631951 18170 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:04:43.634660 18185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:43.636127 18170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:04:43.636318 18170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "b089d29fda264c4a88cdb082163ac8d4"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "46e5eb33e158988733c7e28dd2020ce5"
server_key_iv: "d8e986b603b1ff583b4244d03495d851"
server_key_version: "encryptionkey@0"
I20260619 13:04:43.637290 18170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:04:43.660212 18170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:04:43.664310 18170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:04:43.664583 18170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:04:43.665562 18170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:04:43.666917 18170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:04:43.667038 18170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:43.667176 18170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:04:43.667266 18170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:43.700937 18170 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:38229
I20260619 13:04:43.701021 18297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:38229 every 8 connection(s)
I20260619 13:04:43.702337 18170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:04:43.704504 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 18170
I20260619 13:04:43.705111 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:36423
--local_ip_for_outbound_sockets=127.1.87.67
--tserver_master_addrs=127.1.87.126:33205
--webserver_port=46451
--webserver_interface=127.1.87.67
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:04:43.707868 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:43.705566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:35303)
0619 13:04:43.706433 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:43.706695 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":309,"threads_started":1}
W20260619 13:04:43.708577 18298 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:33205 (0 consecutive failures): Network error: Failed to ping master at 127.1.87.126:33205: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
I20260619 13:04:43.709530 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:43.709080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:45797)
0619 13:04:43.709263 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:43.709376 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 13:04:43.710564 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:43.710131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:47109)
0619 13:04:43.710309 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:43.710411 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 13:04:43.710858 18298 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:04:44.027486 18301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:04:44.027854 18301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:04:44.028055 18301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:04:44.036049 18301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:04:44.036175 18301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:04:44.036291 18301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:04:44.046360 18301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:36423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=46451
--tserver_master_addrs=127.1.87.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:04:44.047704 18301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:04:44.048966 18301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:04:44.062318 18307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:44.062357 18309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:04:44.062994 18306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:04:44.063809 18301 server_base.cc:1034] running on GCE node
I20260619 13:04:44.064525 18301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:04:44.065569 18301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:04:44.066851 18301 hybrid_clock.cc:648] HybridClock initialized: now 1781874284066813 us; error 71 us; skew 500 ppm
I20260619 13:04:44.067188 18301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:04:44 dist-test-slave-tpsm krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874284, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:04:44.070598 18301 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:04:44.072268 18301 webserver.cc:460] Webserver started at http://127.1.87.67:46451/ using document root <none> and password file <none>
I20260619 13:04:44.072990 18301 fs_manager.cc:359] Metadata directory not provided
I20260619 13:04:44.073185 18301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:04:44.079954 18301 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:04:44.082934 18316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:44.084513 18301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:04:44.084725 18301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "b3464efc4ba74b31b68c23dd72959ad9"
format_stamp: "Formatted at 2026-06-19 13:03:57 on dist-test-slave-tpsm"
server_key: "b46ad94f393166f7749877c23d1f7e8c"
server_key_iv: "1e3e42f28cff1afa5e605ec77798b9bd"
server_key_version: "encryptionkey@0"
I20260619 13:04:44.085762 18301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:04:44.118531 18301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:04:44.124308 18301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:04:44.124624 18301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:04:44.125738 18301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:04:44.127151 18301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:04:44.127271 18301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:44.127426 18301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:04:44.127521 18301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:04:44.162535 18301 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:36423
I20260619 13:04:44.162590 18428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:36423 every 8 connection(s)
I20260619 13:04:44.164054 18301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:04:44.169020 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:44.167235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:37613)
0619 13:04:44.168008 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:44.168210 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":281,"threads_started":1}
W20260619 13:04:44.169670 18429 heartbeater.cc:644] Failed to heartbeat to 127.1.87.126:33205 (0 consecutive failures): Network error: Failed to ping master at 127.1.87.126:33205: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
I20260619 13:04:44.170462 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:44.170151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:45259)
0619 13:04:44.170291 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:44.170363 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 13:04:44.170940 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 18301
I20260619 13:04:44.172262 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:44.171819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:33741)
0619 13:04:44.171997 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:44.172102 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 13:04:44.172605 18429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 13:04:44.231860 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:44.231032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:55197)
0619 13:04:44.231496 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:44.231571 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":199,"threads_started":1}
I20260619 13:04:44.712365 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:44.711551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:45075)
0619 13:04:44.712036 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:44.712125 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":247,"threads_started":1}
I20260619 13:04:45.174089 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:45.173206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:54601)
0619 13:04:45.173755 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:45.173843 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":361,"threads_started":1}
I20260619 13:04:45.233667 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:45.232900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:35633)
0619 13:04:45.233383 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:45.233489 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":259,"threads_started":1}
I20260619 13:04:45.714085 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:45.713302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:54357)
0619 13:04:45.713839 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:45.713919 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":248,"threads_started":1}
I20260619 13:04:46.175688 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:46.174934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:43643)
0619 13:04:46.175441 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:46.175519 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260619 13:04:46.235270 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:46.234600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:41495)
0619 13:04:46.235020 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:46.235111 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":221,"threads_started":1}
I20260619 13:04:46.715961 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:46.715184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:41815)
0619 13:04:46.715694 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:46.715778 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":293,"threads_started":1}
I20260619 13:04:47.177263 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:47.176625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:47587)
0619 13:04:47.177026 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:47.177101 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":230,"threads_started":1}
I20260619 13:04:47.236984 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:47.236234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:34055)
0619 13:04:47.236705 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:47.236799 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":254,"threads_started":1}
I20260619 13:04:47.718400 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:47.717301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:48771)
0619 13:04:47.718082 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:47.718187 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":417,"threads_started":1}
I20260619 13:04:48.179464 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:48.178375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:37005)
0619 13:04:48.179037 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:48.179142 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":345,"threads_started":1}
I20260619 13:04:48.238829 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:48.238080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:38387)
0619 13:04:48.238576 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:48.238664 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":260,"threads_started":1}
I20260619 13:04:48.720290 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:48.719493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:45241)
0619 13:04:48.720022 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:48.720116 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":296,"threads_started":1}
I20260619 13:04:49.181286 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:49.180538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:54659)
0619 13:04:49.181025 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:49.181108 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":259,"threads_started":1}
I20260619 13:04:49.240629 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:49.239851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:58559)
0619 13:04:49.240364 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:49.240456 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":268,"threads_started":1}
I20260619 13:04:49.722143 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:49.721267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:46043)
0619 13:04:49.721802 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:49.721898 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":276,"threads_started":1}
I20260619 13:04:50.183055 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:50.182275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:48513)
0619 13:04:50.182791 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:50.182889 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":279,"threads_started":1}
I20260619 13:04:50.242482 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:50.241631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:43029)
0619 13:04:50.242112 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:50.242269 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":224,"threads_started":1}
I20260619 13:04:50.723999 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:50.723104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:50949)
0619 13:04:50.723699 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:50.723784 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":276,"threads_started":1}
I20260619 13:04:51.184935 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:51.184078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:57593)
0619 13:04:51.184611 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:51.184709 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":227,"threads_started":1}
I20260619 13:04:51.244336 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:51.243492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:35205)
0619 13:04:51.244001 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:51.244112 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260619 13:04:51.725791 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:51.724968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:36101)
0619 13:04:51.725521 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:51.725624 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":254,"threads_started":1}
I20260619 13:04:52.186702 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:52.185973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:49271)
0619 13:04:52.186458 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:52.186542 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":260,"threads_started":1}
I20260619 13:04:52.246085 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:52.245358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:51911)
0619 13:04:52.245828 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:52.245926 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":230,"threads_started":1}
I20260619 13:04:52.727588 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:52.726721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:51585)
0619 13:04:52.727276 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:52.727404 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":307,"threads_started":1}
I20260619 13:04:53.188664 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:53.187796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:44953)
0619 13:04:53.188372 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:53.188488 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":263,"threads_started":1}
I20260619 13:04:53.247853 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:53.247015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:48823)
0619 13:04:53.247574 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:53.247656 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":227,"threads_started":1}
I20260619 13:04:53.729233 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:53.728539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:55009)
0619 13:04:53.728988 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:53.729069 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
I20260619 13:04:54.190517 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:54.189697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:52351)
0619 13:04:54.190248 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:54.190338 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":253,"threads_started":1}
I20260619 13:04:54.249485 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:54.248754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:36627)
0619 13:04:54.249201 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:54.249301 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":250,"threads_started":1}
I20260619 13:04:54.730962 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:54.730227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:55215)
0619 13:04:54.730678 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:54.730784 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
I20260619 13:04:55.192132 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:55.191428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:54571)
0619 13:04:55.191861 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:55.191947 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":222,"threads_started":1}
I20260619 13:04:55.251140 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:55.250425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:39591)
0619 13:04:55.250882 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:55.250963 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":251,"threads_started":1}
I20260619 13:04:55.732625 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:55.731911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:48563)
0619 13:04:55.732381 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:55.732462 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":220,"threads_started":1}
I20260619 13:04:56.194442 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:56.193490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:37367)
0619 13:04:56.194064 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:56.194179 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
I20260619 13:04:56.253221 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:56.252422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:55819)
0619 13:04:56.252901 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:56.253010 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":215,"threads_started":1}
I20260619 13:04:56.734516 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:56.733775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:38401)
0619 13:04:56.734239 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:56.734332 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":222,"threads_started":1}
I20260619 13:04:57.196187 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:57.195432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:56633)
0619 13:04:57.195894 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:57.195986 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":230,"threads_started":1}
I20260619 13:04:57.255769 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:57.254262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:39941)
0619 13:04:57.255290 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:57.255548 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":895,"thread_start_us":711,"threads_started":1}
I20260619 13:04:57.736238 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:57.735526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:52567)
0619 13:04:57.735978 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:57.736059 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":234,"threads_started":1}
I20260619 13:04:58.197913 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:58.197124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:43435)
0619 13:04:58.197627 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:58.197718 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":253,"threads_started":1}
I20260619 13:04:58.257687 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:58.256950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.65:53955)
0619 13:04:58.257439 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:58.257523 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":278,"threads_started":1}
I20260619 13:04:58.738178 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:58.737386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.66:40371)
0619 13:04:58.737871 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:58.737995 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":248,"threads_started":1}
I20260619 13:04:59.175614 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 17998
I20260619 13:04:59.197938 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:59.176973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.1.87.65:39873)
0619 13:04:59.177748 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:59.177770 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:59.177828 (+ 58us) server_negotiation.cc:407] Connection header received
0619 13:04:59.179926 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:59.179936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:59.180384 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:59.180763 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:59.182284 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:59.183696 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:59.185051 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:59.185884 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:59.186038 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:59.191649 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:59.191658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:59.191675 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:59.191719 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:59.194549 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:59.195221 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:59.195228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:59.195234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:59.195496 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:59.195859 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:59.195862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:59.195865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:59.196435 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:59.196667 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:59.197136 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 13:04:59.197606 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":385,"threads_started":1}
I20260619 13:04:59.199469 18162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:04:59.200002 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:59.199042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:33205 (local address 127.1.87.67:36761)
0619 13:04:59.199575 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 13:04:59.199715 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.126:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":288,"threads_started":1}
I20260619 13:04:59.289021 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 17998
I20260619 13:04:59.308331 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 18170
I20260619 13:04:59.328896 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:59.309527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.1.87.66:38229)
0619 13:04:59.310138 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:59.310152 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:59.310190 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:04:59.311887 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:59.311897 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:59.312324 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:59.312643 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:59.314603 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:59.316617 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:59.318075 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:59.318852 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:59.319000 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:59.322156 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:59.322165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:59.322183 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:59.322242 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:59.325539 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:59.326308 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:59.326315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:59.326322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:59.326518 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:59.326897 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:59.326901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:59.326903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:59.327572 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:59.327809 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:59.328225 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 13:04:59.328596 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":330,"threads_started":1}
I20260619 13:04:59.330926 18293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:04:59.420794 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 18170
I20260619 13:04:59.439477 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 18301
I20260619 13:04:59.457860 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:04:59.440617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.1.87.67:36423)
0619 13:04:59.441240 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 13:04:59.441254 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:04:59.441294 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:04:59.443087 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:04:59.443097 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:04:59.443666 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:04:59.443963 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:04:59.445374 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:59.446582 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:59.447787 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:04:59.448415 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:04:59.448550 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:04:59.451133 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:04:59.451141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:04:59.451155 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:04:59.451202 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:04:59.454402 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:59.455274 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:59.455281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:59.455287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:59.455539 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:04:59.455950 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:04:59.455953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:04:59.455955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:04:59.456501 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:04:59.456753 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 13:04:59.457106 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 13:04:59.457541 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":302,"threads_started":1}
I20260619 13:04:59.459425 18424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:04:59.545194 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 18301
2026-06-19T13:04:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65043 ms)
[----------] 30 tests from SecurityITest (295037 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:04:59 dist-test-slave-tpsm krb5kdc[18486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:04:59 dist-test-slave-tpsm krb5kdc[18486](info): set up 2 sockets
Jun 19 13:04:59 dist-test-slave-tpsm krb5kdc[18486](info): commencing operation
krb5kdc: starting...
W20260619 13:05:00.659588 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:00 dist-test-slave-tpsm krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:00.904923 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44271
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:44271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:01.237600 18499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:01.237893 18499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:01.238058 18499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:01.245296 18499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:01.245409 18499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:01.245502 18499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:01.245620 18499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:01.245682 18499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:01.255452 18499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:44271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:01.256801 18499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:01.258085 18499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:01.269660 18505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:01.269690 18504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:01.269749 18507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:01.270582 18499 server_base.cc:1034] running on GCE node
I20260619 13:05:01.271399 18499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:01.272786 18499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:01.273985 18499 hybrid_clock.cc:648] HybridClock initialized: now 1781874301273951 us; error 32 us; skew 500 ppm
I20260619 13:05:01.274335 18499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:01 dist-test-slave-tpsm krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:01.277514 18499 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:01.279102 18499 webserver.cc:460] Webserver started at http://127.1.87.126:46375/ using document root <none> and password file <none>
I20260619 13:05:01.279856 18499 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:01.279984 18499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:01.280301 18499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:01.283209 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "15db6e187faf4a89b7281d24ed6b6fd7"
format_stamp: "Formatted at 2026-06-19 13:05:01 on dist-test-slave-tpsm"
server_key: "4f970c6c22fa86e77a5753a11c1b4af3"
server_key_iv: "16a84052a8ea87a5ca7973b27e4af7dd"
server_key_version: "encryptionkey@0"
I20260619 13:05:01.284032 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "15db6e187faf4a89b7281d24ed6b6fd7"
format_stamp: "Formatted at 2026-06-19 13:05:01 on dist-test-slave-tpsm"
server_key: "4f970c6c22fa86e77a5753a11c1b4af3"
server_key_iv: "16a84052a8ea87a5ca7973b27e4af7dd"
server_key_version: "encryptionkey@0"
I20260619 13:05:01.289191 18499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:05:01.292668 18514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:01.293913 18499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:01.294092 18499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "15db6e187faf4a89b7281d24ed6b6fd7"
format_stamp: "Formatted at 2026-06-19 13:05:01 on dist-test-slave-tpsm"
server_key: "4f970c6c22fa86e77a5753a11c1b4af3"
server_key_iv: "16a84052a8ea87a5ca7973b27e4af7dd"
server_key_version: "encryptionkey@0"
I20260619 13:05:01.294291 18499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:01.309947 18499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:01.314069 18499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:01.314352 18499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:01.332759 18499 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:44271
I20260619 13:05:01.332795 18565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:44271 every 8 connection(s)
I20260619 13:05:01.334136 18499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:01.337791 18566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:01.340106 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 18499
I20260619 13:05:01.340329 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:01.341154 1373 external_mini_cluster.cc:1428] Setting key 65bd264608d0accd507d798b363160d9
I20260619 13:05:01.350433 18566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7: Bootstrap starting.
I20260619 13:05:01.354151 18566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:01.355504 18566 log.cc:824] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:05:01 dist-test-slave-tpsm krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:01.358376 18566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7: No bootstrap required, opened a new log
I20260619 13:05:01.364440 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:01.343218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.1.87.126:44271)
0619 13:05:01.344095 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:01.344106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:01.344152 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:05:01.345798 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:01.345817 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:01.346158 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:01.346464 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:01.348229 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:01.349547 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:01.351460 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:01.352103 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:01.352230 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:01.357256 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:01.357264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:01.357278 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:01.357310 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:01.360476 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:01.361320 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:01.361325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:01.361331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:01.361493 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:01.361879 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:01.361885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:01.361887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:01.362373 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:01.362582 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:01.363038 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 13:05:01.363458 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":377,"threads_started":1}
I20260619 13:05:01.366948 18566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15db6e187faf4a89b7281d24ed6b6fd7" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44271 } }
I20260619 13:05:01.367285 18566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:01.367440 18566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15db6e187faf4a89b7281d24ed6b6fd7, State: Initialized, Role: FOLLOWER
I20260619 13:05:01.367977 18566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15db6e187faf4a89b7281d24ed6b6fd7" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44271 } }
I20260619 13:05:01.368209 18566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:01.368364 18566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:01.368522 18566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:01.370632 18566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15db6e187faf4a89b7281d24ed6b6fd7" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44271 } }
I20260619 13:05:01.371174 18566 leader_election.cc:302] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15db6e187faf4a89b7281d24ed6b6fd7; no voters:
I20260619 13:05:01.371766 18566 leader_election.cc:288] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:01.371891 18571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:01.372613 18571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [term 1 LEADER]: Becoming Leader. State: Replica: 15db6e187faf4a89b7281d24ed6b6fd7, State: Running, Role: LEADER
I20260619 13:05:01.373126 18571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15db6e187faf4a89b7281d24ed6b6fd7" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44271 } }
I20260619 13:05:01.373795 18566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:01.375980 18573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15db6e187faf4a89b7281d24ed6b6fd7. Latest consensus state: current_term: 1 leader_uuid: "15db6e187faf4a89b7281d24ed6b6fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15db6e187faf4a89b7281d24ed6b6fd7" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44271 } } }
I20260619 13:05:01.375953 18572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15db6e187faf4a89b7281d24ed6b6fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15db6e187faf4a89b7281d24ed6b6fd7" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44271 } } }
I20260619 13:05:01.376398 18573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:01.376463 18572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:01.378021 18576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:01.383737 18576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:01.395190 18576 catalog_manager.cc:1347] Generated new cluster ID: 0f253d0c696341599585f47c738fc714
I20260619 13:05:01.395355 18576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:01.418949 18576 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:01.420377 18576 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:01.429809 18576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7: Generated new TSK 0
I20260619 13:05:01.430687 18576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:05:01.490110 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44271
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:01.832243 18592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:01.832691 18592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:01.832805 18592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:01.840689 18592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:01.840827 18592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:01.840945 18592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:05:01.850867 18592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:01.852211 18592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:01.853446 18592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:01.866083 18597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:01.866284 18598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:01.866276 18600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:01.867564 18592 server_base.cc:1034] running on GCE node
I20260619 13:05:01.868127 18592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:01.868988 18592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:01.870224 18592 hybrid_clock.cc:648] HybridClock initialized: now 1781874301870145 us; error 63 us; skew 500 ppm
I20260619 13:05:01.870517 18592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:01 dist-test-slave-tpsm krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:01.873859 18592 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:05:01.876370 18592 webserver.cc:460] Webserver started at http://127.1.87.65:37235/ using document root <none> and password file <none>
I20260619 13:05:01.877144 18592 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:01.877285 18592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:01.877630 18592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:01.880893 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "3eef768003b0452eae8fd6cae22da153"
format_stamp: "Formatted at 2026-06-19 13:05:01 on dist-test-slave-tpsm"
server_key: "62a9d5742a63e95ea248f6ae710abb15"
server_key_iv: "38e3a2d435c71dda381e244f40ee41a6"
server_key_version: "encryptionkey@0"
I20260619 13:05:01.881707 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "3eef768003b0452eae8fd6cae22da153"
format_stamp: "Formatted at 2026-06-19 13:05:01 on dist-test-slave-tpsm"
server_key: "62a9d5742a63e95ea248f6ae710abb15"
server_key_iv: "38e3a2d435c71dda381e244f40ee41a6"
server_key_version: "encryptionkey@0"
I20260619 13:05:01.887033 18592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:05:01.890858 18607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:01.892442 18592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:01.892633 18592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "3eef768003b0452eae8fd6cae22da153"
format_stamp: "Formatted at 2026-06-19 13:05:01 on dist-test-slave-tpsm"
server_key: "62a9d5742a63e95ea248f6ae710abb15"
server_key_iv: "38e3a2d435c71dda381e244f40ee41a6"
server_key_version: "encryptionkey@0"
I20260619 13:05:01.892844 18592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:01.924325 18592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:01.928546 18592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:01.928859 18592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:01.929876 18592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:01.931277 18592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:01.931412 18592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:01.931520 18592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:01.931566 18592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:01.966748 18592 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:33301
I20260619 13:05:01.966856 18719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:33301 every 8 connection(s)
I20260619 13:05:01.968228 18592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:05:01.974870 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 18592
I20260619 13:05:01.975107 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:05:01.975957 1373 external_mini_cluster.cc:1428] Setting key 4883ff5e0049c3748862dc845b20913f
Jun 19 13:05:01 dist-test-slave-tpsm krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:01.993875 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:01.970873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:38079 (local address 127.1.87.126:44271)
0619 13:05:01.971390 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:01.971395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:01.972252 (+ 857us) server_negotiation.cc:407] Connection header received
0619 13:05:01.974596 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:01.974600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:01.974708 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:01.974804 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:01.977034 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:01.977841 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:01.980358 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:01.981001 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:01.981114 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:01.986688 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:01.986696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:01.986704 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:01.986751 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:01.990306 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:01.991140 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:01.991149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:01.991155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:01.991338 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:01.991860 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:01.991868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:01.991874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:01.992261 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:01.992413 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:01.993133 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 13:05:01.993389 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":254,"threads_started":1}
I20260619 13:05:01.994506 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:01.971290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44271 (local address 127.1.87.65:38079)
0619 13:05:01.972080 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:01.972124 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:01.974094 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:01.975098 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:01.975110 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:01.975911 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:01.976813 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:01.976835 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:01.978058 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:01.978063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:01.980183 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:01.980203 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:01.981257 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:01.981267 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:01.981588 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:01.982581 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:01.982623 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:01.986352 (+ 3729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:01.990493 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:01.990510 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:01.990525 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:01.990984 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:01.991494 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:01.991498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:01.991501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:01.991717 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:01.992384 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:01.992390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:01.992730 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 13:05:01.992961 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 13:05:01.993359 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":326,"threads_started":1}
I20260619 13:05:01.996202 18720 heartbeater.cc:347] Connected to a master server at 127.1.87.126:44271
I20260619 13:05:01.996505 18720 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:01.997339 18720 heartbeater.cc:511] Master 127.1.87.126:44271 requested a full tablet report, sending...
I20260619 13:05:01.999770 18531 ts_manager.cc:194] Registered new tserver with Master: 3eef768003b0452eae8fd6cae22da153 (127.1.87.65:33301)
I20260619 13:05:02.002100 18531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:38079
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:05:02.034572 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44271
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:02.364074 18726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:02.364338 18726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:02.364495 18726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:02.371599 18726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:02.371724 18726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:02.371909 18726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:05:02.382014 18726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:02.383239 18726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:02.384502 18726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:02.397363 18731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:02.397298 18734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:02.397317 18732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:02.398211 18726 server_base.cc:1034] running on GCE node
I20260619 13:05:02.398813 18726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:02.399806 18726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:02.401037 18726 hybrid_clock.cc:648] HybridClock initialized: now 1781874302400985 us; error 45 us; skew 500 ppm
I20260619 13:05:02.401356 18726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:02 dist-test-slave-tpsm krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874302, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:02.404546 18726 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:05:02.405934 18726 webserver.cc:460] Webserver started at http://127.1.87.66:39475/ using document root <none> and password file <none>
I20260619 13:05:02.406621 18726 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:02.406746 18726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:02.407063 18726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:02.410089 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "0e90e7b62e124ba2a8dd89329aaf9e72"
format_stamp: "Formatted at 2026-06-19 13:05:02 on dist-test-slave-tpsm"
server_key: "e11ef19aa0acb16ed1fabd4008af806c"
server_key_iv: "c937d89faa9f5455f3ad7169507fdeee"
server_key_version: "encryptionkey@0"
I20260619 13:05:02.410868 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "0e90e7b62e124ba2a8dd89329aaf9e72"
format_stamp: "Formatted at 2026-06-19 13:05:02 on dist-test-slave-tpsm"
server_key: "e11ef19aa0acb16ed1fabd4008af806c"
server_key_iv: "c937d89faa9f5455f3ad7169507fdeee"
server_key_version: "encryptionkey@0"
I20260619 13:05:02.415963 18726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 13:05:02.419476 18741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:02.420774 18726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 13:05:02.420967 18726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "0e90e7b62e124ba2a8dd89329aaf9e72"
format_stamp: "Formatted at 2026-06-19 13:05:02 on dist-test-slave-tpsm"
server_key: "e11ef19aa0acb16ed1fabd4008af806c"
server_key_iv: "c937d89faa9f5455f3ad7169507fdeee"
server_key_version: "encryptionkey@0"
I20260619 13:05:02.421172 18726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:02.437281 18726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:02.441262 18726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:02.441541 18726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:02.442492 18726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:02.443852 18726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:02.443928 18726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:02.444113 18726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:02.444173 18726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:02.486716 18726 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:35203
I20260619 13:05:02.486757 18853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:35203 every 8 connection(s)
I20260619 13:05:02.488104 18726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:05:02.490320 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 18726
I20260619 13:05:02.490523 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:05:02.491083 1373 external_mini_cluster.cc:1428] Setting key cb34dbb08a869b44fbd0976a2285aa46
Jun 19 13:05:02 dist-test-slave-tpsm krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874302, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:02.510185 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:02.490775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:57127 (local address 127.1.87.126:44271)
0619 13:05:02.490956 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:02.490960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:02.492448 (+ 1488us) server_negotiation.cc:407] Connection header received
0619 13:05:02.494708 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:02.494713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:02.494800 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:02.494923 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:02.496897 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:02.497708 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:02.499480 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:02.499929 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:02.500068 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:02.503850 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:02.503854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:02.503858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:02.503890 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:02.506589 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:02.507385 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:02.507389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:02.507391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:02.507518 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:02.507978 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:02.507987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:02.507989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:02.508260 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:02.508412 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:02.509148 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 13:05:02.509624 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:05:02.511262 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:02.491410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44271 (local address 127.1.87.66:57127)
0619 13:05:02.492277 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:02.492322 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:02.494401 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:02.495477 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:02.495486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:02.496079 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:02.496699 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:02.496736 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:02.497880 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:02.497884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:02.499277 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:02.499289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:02.500163 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:02.500171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:02.500411 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:02.501124 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:02.501147 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:02.503596 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:02.506737 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:02.506748 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:02.506767 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:02.507206 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:02.507633 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:02.507639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:02.507643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:02.507847 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:02.508397 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:02.508406 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:02.508765 (+ 359us) client_negotiation.cc:769] Sending connection context
0619 13:05:02.509045 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 13:05:02.510058 (+ 1013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":352,"threads_started":1}
I20260619 13:05:02.513036 18854 heartbeater.cc:347] Connected to a master server at 127.1.87.126:44271
I20260619 13:05:02.513550 18854 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:02.514889 18854 heartbeater.cc:511] Master 127.1.87.126:44271 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
I20260619 13:05:02.516747 18531 ts_manager.cc:194] Registered new tserver with Master: 0e90e7b62e124ba2a8dd89329aaf9e72 (127.1.87.66:35203)
I20260619 13:05:02.517805 18531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:57127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:05:02.546089 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44271
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:02.883394 18859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:02.883683 18859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:02.883782 18859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:02.891090 18859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:02.891201 18859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:02.891340 18859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:05:02.901588 18859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:02.902773 18859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:02.904031 18859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:02.917008 18864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:02.917588 18865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:02.917583 18867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:02.918077 18859 server_base.cc:1034] running on GCE node
I20260619 13:05:02.918726 18859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:02.919823 18859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:02.921139 18859 hybrid_clock.cc:648] HybridClock initialized: now 1781874302921044 us; error 52 us; skew 500 ppm
I20260619 13:05:02.921594 18859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:02 dist-test-slave-tpsm krb5kdc[18486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874302, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:02.925010 18859 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:05:02.926566 18859 webserver.cc:460] Webserver started at http://127.1.87.67:33301/ using document root <none> and password file <none>
I20260619 13:05:02.927241 18859 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:02.927397 18859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:02.927742 18859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:02.930589 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "c39bd91645da4010be90766fc527cb1a"
format_stamp: "Formatted at 2026-06-19 13:05:02 on dist-test-slave-tpsm"
server_key: "af9f5d76e5c487f5deb6622e3ee208df"
server_key_iv: "5a92574925ca47df5b56e94b4babb8af"
server_key_version: "encryptionkey@0"
I20260619 13:05:02.931447 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "c39bd91645da4010be90766fc527cb1a"
format_stamp: "Formatted at 2026-06-19 13:05:02 on dist-test-slave-tpsm"
server_key: "af9f5d76e5c487f5deb6622e3ee208df"
server_key_iv: "5a92574925ca47df5b56e94b4babb8af"
server_key_version: "encryptionkey@0"
I20260619 13:05:02.936476 18859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 13:05:02.939826 18874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:02.941140 18859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:05:02.941350 18859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "c39bd91645da4010be90766fc527cb1a"
format_stamp: "Formatted at 2026-06-19 13:05:02 on dist-test-slave-tpsm"
server_key: "af9f5d76e5c487f5deb6622e3ee208df"
server_key_iv: "5a92574925ca47df5b56e94b4babb8af"
server_key_version: "encryptionkey@0"
I20260619 13:05:02.941587 18859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:02.967087 18859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260619 13:05:02.970523 18716 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.1.87.65:33301
I20260619 13:05:02.971529 18859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:02.971774 18859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:02.972666 18859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:02.973985 18859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:02.974061 18859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:02.974146 18859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:02.974190 18859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:03.005244 18720 heartbeater.cc:503] Master 127.1.87.126:44271 was elected leader, sending a full tablet report...
I20260619 13:05:03.008960 18859 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:44923
I20260619 13:05:03.009045 18986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:44923 every 8 connection(s)
I20260619 13:05:03.010341 18859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:05:03.013881 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 18859
I20260619 13:05:03.014061 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:05:03.014678 1373 external_mini_cluster.cc:1428] Setting key 85b5775ccfeeaddff49c480414c822f5
Jun 19 13:05:03 dist-test-slave-tpsm krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874302, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:03.031926 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.013204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:53997 (local address 127.1.87.126:44271)
0619 13:05:03.013688 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.013694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.015249 (+ 1555us) server_negotiation.cc:407] Connection header received
0619 13:05:03.017163 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.017167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.017239 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.017356 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:03.019103 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.019979 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.021510 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.021912 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.022026 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.025789 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:03.025794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:03.025797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:03.025848 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:03.028817 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.029669 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.029674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.029677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.029839 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.030340 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.030343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.030345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.030620 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:03.030734 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.031476 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.031648 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260619 13:05:03.032874 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.013716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44271 (local address 127.1.87.67:53997)
0619 13:05:03.014609 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:03.014667 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:03.016833 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:03.017614 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:03.017626 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:03.018283 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:03.018941 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.018958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.020145 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.020149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.021364 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.021376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.022097 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.022101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.022288 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.022971 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:03.022992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:03.025547 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:03.028951 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:03.028960 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:03.028977 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:03.029539 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:03.029957 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:03.029962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:03.029964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:03.030213 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:03.030757 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:03.030763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:03.031074 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 13:05:03.031372 (+ 298us) client_negotiation.cc:240] Negotiation successful
0619 13:05:03.031779 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":380,"threads_started":1}
I20260619 13:05:03.034281 18987 heartbeater.cc:347] Connected to a master server at 127.1.87.126:44271
I20260619 13:05:03.034617 18987 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:03.035516 18987 heartbeater.cc:511] Master 127.1.87.126:44271 requested a full tablet report, sending...
I20260619 13:05:03.036942 18531 ts_manager.cc:194] Registered new tserver with Master: c39bd91645da4010be90766fc527cb1a (127.1.87.67:44923)
I20260619 13:05:03.037743 18531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:53997
I20260619 13:05:03.042507 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:05:03.059597 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.047003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.1.87.126:44271)
0619 13:05:03.047162 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.047166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.047683 (+ 517us) server_negotiation.cc:407] Connection header received
0619 13:05:03.047823 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.047826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.047884 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.047964 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:03.049325 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.050004 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.051381 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.051785 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.051880 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.054304 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:03.054308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:03.054311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:03.054342 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:03.056850 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.057506 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.057510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.057513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.057667 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.058079 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.058083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.058085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.058427 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:03.058623 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.058974 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.059149 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:05:03.063983 18531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:05:03.071527 18531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:05:03.094928 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.084328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.1.87.66:35203)
0619 13:05:03.086518 (+ 2190us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.086531 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.086563 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:05:03.086687 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.086698 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.086991 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.087569 (+ 578us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:03.088479 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.089996 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.092054 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.093603 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.093734 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.093929 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.094425 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.094585 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1987,"thread_start_us":2127,"threads_started":1}
I20260619 13:05:03.094986 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.084171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:35203 (local address 127.0.0.1:50256)
0619 13:05:03.085018 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:03.085072 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:03.085242 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:03.087287 (+ 2045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:03.087298 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:03.087861 (+ 563us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:03.088294 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.088306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.090125 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.090132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.091876 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.091889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.093818 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.093825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.094171 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.094233 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 13:05:03.094367 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 13:05:03.094556 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":45,"spinlock_wait_cycles":1135616,"thread_start_us":1432,"threads_started":1}
I20260619 13:05:03.097019 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.084279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:44923 (local address 127.0.0.1:48776)
0619 13:05:03.085940 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:03.085963 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:03.087717 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:03.089072 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:03.089079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:03.089108 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:03.089595 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.089607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.091576 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.091580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.094052 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.094064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.096148 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.096153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.096304 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.096367 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 13:05:03.096451 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 13:05:03.096587 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"mutex_wait_us":129,"thread_start_us":843,"threads_started":1}
I20260619 13:05:03.097111 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.084892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.1.87.67:44923)
0619 13:05:03.088282 (+ 3390us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.088288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.088310 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:05:03.088376 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.088381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.088610 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.088789 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:03.089776 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.091307 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.094211 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.095683 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.095812 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.096078 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.096521 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.096706 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3260,"thread_start_us":3093,"threads_started":1}
I20260619 13:05:03.098212 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.086503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.1.87.65:33301)
0619 13:05:03.087186 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.087197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.087864 (+ 667us) server_negotiation.cc:407] Connection header received
0619 13:05:03.088155 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.088164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.088384 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.088562 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:03.089605 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.091220 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.094294 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.095684 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.095812 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.096023 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.097690 (+ 1667us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.097854 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":337,"threads_started":1}
I20260619 13:05:03.098161 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.085950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:33301 (local address 127.0.0.1:37138)
0619 13:05:03.087768 (+ 1818us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:03.087786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:03.087960 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:03.088844 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:03.088848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:03.088872 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:03.089404 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.089420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.091436 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.091442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.094159 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.094172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.095872 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.095877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.097498 (+ 1621us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.097537 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 13:05:03.097646 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 13:05:03.097792 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1707,"mutex_wait_us":214,"thread_start_us":1580,"threads_started":1}
I20260619 13:05:03.099555 18789 tablet_service.cc:1467] Processing CreateTablet for tablet 56d7d0cf590541289ede813fbb49e3a2 (DEFAULT_TABLE table=test-table [id=953da8f09ce04189a7fe8cab0c7a966d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:03.100729 18789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d7d0cf590541289ede813fbb49e3a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:03.101127 18922 tablet_service.cc:1467] Processing CreateTablet for tablet 56d7d0cf590541289ede813fbb49e3a2 (DEFAULT_TABLE table=test-table [id=953da8f09ce04189a7fe8cab0c7a966d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:03.102385 18922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d7d0cf590541289ede813fbb49e3a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:03.106709 18655 tablet_service.cc:1467] Processing CreateTablet for tablet 56d7d0cf590541289ede813fbb49e3a2 (DEFAULT_TABLE table=test-table [id=953da8f09ce04189a7fe8cab0c7a966d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:03.108129 18655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d7d0cf590541289ede813fbb49e3a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:03.113422 19006 tablet_bootstrap.cc:492] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: Bootstrap starting.
I20260619 13:05:03.116047 19007 tablet_bootstrap.cc:492] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: Bootstrap starting.
I20260619 13:05:03.118307 19006 tablet_bootstrap.cc:654] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:03.119478 19007 tablet_bootstrap.cc:654] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:03.119779 19006 log.cc:824] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:03.120656 19007 log.cc:824] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:03.122146 19010 tablet_bootstrap.cc:492] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: Bootstrap starting.
I20260619 13:05:03.123384 19006 tablet_bootstrap.cc:492] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: No bootstrap required, opened a new log
I20260619 13:05:03.123696 19006 ts_tablet_manager.cc:1397] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 13:05:03.125437 19010 tablet_bootstrap.cc:654] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:03.125993 19007 tablet_bootstrap.cc:492] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: No bootstrap required, opened a new log
I20260619 13:05:03.126262 19007 ts_tablet_manager.cc:1397] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260619 13:05:03.126530 19010 log.cc:824] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:03.128890 19010 tablet_bootstrap.cc:492] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: No bootstrap required, opened a new log
I20260619 13:05:03.129142 19010 ts_tablet_manager.cc:1397] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 13:05:03.135243 19006 raft_consensus.cc:348] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } }
I20260619 13:05:03.135720 19006 raft_consensus.cc:374] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:03.135823 19006 raft_consensus.cc:729] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bd91645da4010be90766fc527cb1a, State: Initialized, Role: FOLLOWER
I20260619 13:05:03.136472 19010 raft_consensus.cc:348] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } }
I20260619 13:05:03.136467 19006 consensus_queue.cc:260] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } }
I20260619 13:05:03.136821 19010 raft_consensus.cc:374] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:03.136634 19007 raft_consensus.cc:348] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } }
I20260619 13:05:03.136920 19010 raft_consensus.cc:729] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eef768003b0452eae8fd6cae22da153, State: Initialized, Role: FOLLOWER
I20260619 13:05:03.137038 19007 raft_consensus.cc:374] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:03.137177 19007 raft_consensus.cc:729] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e90e7b62e124ba2a8dd89329aaf9e72, State: Initialized, Role: FOLLOWER
I20260619 13:05:03.137611 19010 consensus_queue.cc:260] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } }
I20260619 13:05:03.137733 19007 consensus_queue.cc:260] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } }
I20260619 13:05:03.137965 18987 heartbeater.cc:503] Master 127.1.87.126:44271 was elected leader, sending a full tablet report...
I20260619 13:05:03.138504 19006 ts_tablet_manager.cc:1428] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260619 13:05:03.140324 18854 heartbeater.cc:503] Master 127.1.87.126:44271 was elected leader, sending a full tablet report...
I20260619 13:05:03.140952 19007 ts_tablet_manager.cc:1428] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260619 13:05:03.142558 19010 ts_tablet_manager.cc:1428] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260619 13:05:03.221163 18721 tablet.cc:2369] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:05:03.240259 18855 tablet.cc:2369] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:05:03.262247 18988 tablet.cc:2369] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:05:03.305444 19014 raft_consensus.cc:482] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:05:03.305779 19014 raft_consensus.cc:504] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } }
I20260619 13:05:03.307547 19014 leader_election.cc:288] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c39bd91645da4010be90766fc527cb1a (127.1.87.67:44923), 3eef768003b0452eae8fd6cae22da153 (127.1.87.65:33301)
I20260619 13:05:03.316556 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.308592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:53967 (local address 127.1.87.65:33301)
0619 13:05:03.308842 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.308848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.309246 (+ 398us) server_negotiation.cc:407] Connection header received
0619 13:05:03.309412 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.309416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.309473 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.309550 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:03.310151 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.311195 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.314349 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.315615 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.315754 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.315860 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.316168 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.316305 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 13:05:03.316675 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.308122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:33301 (local address 127.1.87.66:53967)
0619 13:05:03.309140 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:03.309157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:03.309272 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:03.309593 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:03.309599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:03.309622 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:03.310008 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.310016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.311366 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.311371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.314152 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.314169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.315857 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.315862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.315995 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.316024 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 13:05:03.316100 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 13:05:03.316218 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":193,"thread_start_us":684,"threads_started":1}
I20260619 13:05:03.317022 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.308075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:53847 (local address 127.1.87.67:44923)
0619 13:05:03.308306 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.308313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.308654 (+ 341us) server_negotiation.cc:407] Connection header received
0619 13:05:03.308941 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.308949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.309029 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.309100 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:03.309945 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.311091 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.314345 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.315627 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.315754 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.315861 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.316415 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.316553 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 13:05:03.317132 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.307747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:44923 (local address 127.1.87.66:53847)
0619 13:05:03.308534 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:03.308551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:03.308732 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:03.309329 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:03.309334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:03.309361 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:03.309789 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.309804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.311245 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.311281 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.314152 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:03.314169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.316023 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.316028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:03.316226 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.316265 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 13:05:03.316364 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 13:05:03.316490 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":165,"thread_start_us":438,"threads_started":1}
I20260619 13:05:03.317776 18675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d7d0cf590541289ede813fbb49e3a2" candidate_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eef768003b0452eae8fd6cae22da153" is_pre_election: true
I20260619 13:05:03.318058 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d7d0cf590541289ede813fbb49e3a2" candidate_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bd91645da4010be90766fc527cb1a" is_pre_election: true
I20260619 13:05:03.318328 18675 raft_consensus.cc:2454] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e90e7b62e124ba2a8dd89329aaf9e72 in term 0.
I20260619 13:05:03.318564 18942 raft_consensus.cc:2454] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e90e7b62e124ba2a8dd89329aaf9e72 in term 0.
I20260619 13:05:03.319176 18745 leader_election.cc:302] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e90e7b62e124ba2a8dd89329aaf9e72, 3eef768003b0452eae8fd6cae22da153; no voters:
I20260619 13:05:03.319624 19014 raft_consensus.cc:2781] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:05:03.319773 19014 raft_consensus.cc:482] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:05:03.319875 19014 raft_consensus.cc:3037] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:03.321991 19014 raft_consensus.cc:504] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } }
I20260619 13:05:03.322589 19014 leader_election.cc:288] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [CANDIDATE]: Term 1 election: Requested vote from peers c39bd91645da4010be90766fc527cb1a (127.1.87.67:44923), 3eef768003b0452eae8fd6cae22da153 (127.1.87.65:33301)
I20260619 13:05:03.323109 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d7d0cf590541289ede813fbb49e3a2" candidate_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bd91645da4010be90766fc527cb1a"
I20260619 13:05:03.323107 18675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d7d0cf590541289ede813fbb49e3a2" candidate_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eef768003b0452eae8fd6cae22da153"
I20260619 13:05:03.323354 18942 raft_consensus.cc:3037] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:03.323338 18675 raft_consensus.cc:3037] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:03.326561 18942 raft_consensus.cc:2454] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e90e7b62e124ba2a8dd89329aaf9e72 in term 1.
I20260619 13:05:03.326570 18675 raft_consensus.cc:2454] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e90e7b62e124ba2a8dd89329aaf9e72 in term 1.
I20260619 13:05:03.327044 18743 leader_election.cc:302] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e90e7b62e124ba2a8dd89329aaf9e72, c39bd91645da4010be90766fc527cb1a; no voters:
I20260619 13:05:03.327414 19014 raft_consensus.cc:2781] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:03.327869 19014 raft_consensus.cc:686] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 1 LEADER]: Becoming Leader. State: Replica: 0e90e7b62e124ba2a8dd89329aaf9e72, State: Running, Role: LEADER
I20260619 13:05:03.328426 19014 consensus_queue.cc:237] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } }
I20260619 13:05:03.333254 18531 catalog_manager.cc:5495] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e90e7b62e124ba2a8dd89329aaf9e72 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e90e7b62e124ba2a8dd89329aaf9e72" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 } health_report { overall_health: UNKNOWN } } }
I20260619 13:05:03.400717 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.395108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.1.87.66:35203)
0619 13:05:03.395373 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.395380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.395408 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:05:03.395498 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.395501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.395577 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.395765 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:05:03.396145 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.396899 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.398509 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.398953 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.399045 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.399602 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:05:03.400112 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:05:03.400226 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.400327 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.400396 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:05:03.408835 18675 raft_consensus.cc:1261] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [term 1 FOLLOWER]: Refusing update from remote peer 0e90e7b62e124ba2a8dd89329aaf9e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:03.408844 18942 raft_consensus.cc:1261] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [term 1 FOLLOWER]: Refusing update from remote peer 0e90e7b62e124ba2a8dd89329aaf9e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:03.409819 19017 consensus_queue.cc:1035] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eef768003b0452eae8fd6cae22da153" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:03.410187 19014 consensus_queue.cc:1035] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bd91645da4010be90766fc527cb1a" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 44923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:03.422849 19021 mvcc.cc:205] Tried to move back new op lower bound from 7298557146752446464 to 7298557146437570560. Current Snapshot: MvccSnapshot[applied={T|T < 7298557146752446464}]
I20260619 13:05:03.436064 18529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:03.436383 18529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:03.440140 18529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15db6e187faf4a89b7281d24ed6b6fd7: Sending DeleteTablet for 3 replicas of tablet 56d7d0cf590541289ede813fbb49e3a2
I20260619 13:05:03.441407 18922 tablet_service.cc:1514] Processing DeleteTablet for tablet 56d7d0cf590541289ede813fbb49e3a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:03 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:48776
I20260619 13:05:03.441515 18789 tablet_service.cc:1514] Processing DeleteTablet for tablet 56d7d0cf590541289ede813fbb49e3a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:03 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:50256
I20260619 13:05:03.441640 18655 tablet_service.cc:1514] Processing DeleteTablet for tablet 56d7d0cf590541289ede813fbb49e3a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:03 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:37138
I20260619 13:05:03.444774 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 18592
I20260619 13:05:03.454595 19031 tablet_replica.cc:330] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: stopping tablet replica
I20260619 13:05:03.455780 19031 raft_consensus.cc:2229] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:03.456986 19031 raft_consensus.cc:2258] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:03.460409 19031 ts_tablet_manager.cc:1905] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:03.467376 19029 tablet_replica.cc:330] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: stopping tablet replica
I20260619 13:05:03.467911 19029 raft_consensus.cc:2229] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:03.468437 19029 raft_consensus.cc:2258] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:03.468641 19030 tablet_replica.cc:330] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: stopping tablet replica
I20260619 13:05:03.469079 19030 raft_consensus.cc:2229] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:05:03.470172 19030 raft_consensus.cc:2258] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:03.470700 19029 ts_tablet_manager.cc:1905] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:03.472332 19030 ts_tablet_manager.cc:1905] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:05:03 dist-test-slave-tpsm krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:05:03.477885 19031 ts_tablet_manager.cc:1918] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:03.477898 19029 ts_tablet_manager.cc:1918] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:03.478068 19031 log.cc:1196] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/56d7d0cf590541289ede813fbb49e3a2
I20260619 13:05:03.478230 19029 log.cc:1196] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/56d7d0cf590541289ede813fbb49e3a2
I20260619 13:05:03.478641 19031 ts_tablet_manager.cc:1939] T 56d7d0cf590541289ede813fbb49e3a2 P 3eef768003b0452eae8fd6cae22da153: Deleting consensus metadata
I20260619 13:05:03.478844 19029 ts_tablet_manager.cc:1939] T 56d7d0cf590541289ede813fbb49e3a2 P c39bd91645da4010be90766fc527cb1a: Deleting consensus metadata
I20260619 13:05:03.480849 18516 catalog_manager.cc:4841] TS c39bd91645da4010be90766fc527cb1a (127.1.87.67:44923): tablet 56d7d0cf590541289ede813fbb49e3a2 (table test-table [id=953da8f09ce04189a7fe8cab0c7a966d]) successfully deleted
I20260619 13:05:03.482055 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.447400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.1.87.65:33301)
0619 13:05:03.447636 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.447645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.459723 (+ 12078us) server_negotiation.cc:407] Connection header received
0619 13:05:03.460318 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.460328 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.460446 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.460537 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:03.462814 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.464266 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.467948 (+ 3682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.468535 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.468649 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.474329 (+ 5680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:03.474343 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:03.474365 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:03.474417 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:03.478438 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.479420 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.479429 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.479432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.479590 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.480099 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.480104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.480107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.480586 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:03.480865 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.481447 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.481678 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:05:03.483572 18715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:03.485018 19030 ts_tablet_manager.cc:1918] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:03.485246 19030 log.cc:1196] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/56d7d0cf590541289ede813fbb49e3a2
I20260619 13:05:03.485951 19030 ts_tablet_manager.cc:1939] T 56d7d0cf590541289ede813fbb49e3a2 P 0e90e7b62e124ba2a8dd89329aaf9e72: Deleting consensus metadata
I20260619 13:05:03.487586 18518 catalog_manager.cc:4841] TS 0e90e7b62e124ba2a8dd89329aaf9e72 (127.1.87.66:35203): tablet 56d7d0cf590541289ede813fbb49e3a2 (table test-table [id=953da8f09ce04189a7fe8cab0c7a966d]) successfully deleted
I20260619 13:05:03.590874 18518 catalog_manager.cc:4841] TS 3eef768003b0452eae8fd6cae22da153 (127.1.87.65:33301): tablet 56d7d0cf590541289ede813fbb49e3a2 (table test-table [id=953da8f09ce04189a7fe8cab0c7a966d]) successfully deleted
I20260619 13:05:03.591938 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 18592
I20260619 13:05:03.614198 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 18726
Jun 19 13:05:03 dist-test-slave-tpsm krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:05:03.635835 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.615020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.1.87.66:35203)
0619 13:05:03.615685 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.615691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.615712 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:05:03.615986 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.616008 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.616093 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.616206 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:03.618065 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.619104 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.621544 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.622264 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.622394 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.628089 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:03.628109 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:03.628136 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:03.628186 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:03.632565 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.633278 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.633293 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.633296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.633493 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.633864 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.633871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.633874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.634405 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:03.634757 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.635141 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.635387 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417}
I20260619 13:05:03.636600 18849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:03.754513 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 18726
I20260619 13:05:03.775178 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 18859
Jun 19 13:05:03 dist-test-slave-tpsm krb5kdc[18486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:05:03.791213 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:03.776066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.1.87.67:44923)
0619 13:05:03.776260 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:03.776267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:03.776340 (+ 73us) server_negotiation.cc:407] Connection header received
0619 13:05:03.776545 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:03.776551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:03.776630 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:03.776737 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:03.778011 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.778736 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.780129 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:03.780528 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:03.780669 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:03.785208 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:03.785221 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:03.785238 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:03.785282 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:03.788430 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.789139 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.789154 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.789157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.789314 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:03.789680 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:03.789686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:03.789691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:03.790030 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:03.790325 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:03.790646 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 13:05:03.790869 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:05:03.791967 18982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:03.880897 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 18859
I20260619 13:05:03.897501 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 18499
I20260619 13:05:03.898660 18561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:04.009068 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 18499
2026-06-19T13:05:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4465 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T13:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:04Z Disabled control of system clock
I20260619 13:05:04.083737 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44887
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:44887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:05:04.405124 19040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:04.405407 19040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:04.405489 19040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:04.412667 19040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:04.412777 19040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:04.412815 19040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:04.412879 19040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:04.412925 19040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:04.422122 19040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:44887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:04.423393 19040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:04.425086 19040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:04.436774 19045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:04.436733 19046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:04.436746 19048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:04.437733 19040 server_base.cc:1034] running on GCE node
I20260619 13:05:04.438438 19040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:04.439788 19040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:04.440994 19040 hybrid_clock.cc:648] HybridClock initialized: now 1781874304440958 us; error 28 us; skew 500 ppm
I20260619 13:05:04.441299 19040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:04.446027 19040 webserver.cc:460] Webserver started at http://127.1.87.126:37903/ using document root <none> and password file <none>
I20260619 13:05:04.446682 19040 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:04.446767 19040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:04.447018 19040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:04.449854 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "fa058c83055d4074bd6fbaa45b566087"
format_stamp: "Formatted at 2026-06-19 13:05:04 on dist-test-slave-tpsm"
server_key: "e82c4166b6d99d559b27a0526c135b7f"
server_key_iv: "a651c4f8ff679a68af00fb32c7c457b9"
server_key_version: "encryptionkey@0"
I20260619 13:05:04.450630 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "fa058c83055d4074bd6fbaa45b566087"
format_stamp: "Formatted at 2026-06-19 13:05:04 on dist-test-slave-tpsm"
server_key: "e82c4166b6d99d559b27a0526c135b7f"
server_key_iv: "a651c4f8ff679a68af00fb32c7c457b9"
server_key_version: "encryptionkey@0"
I20260619 13:05:04.455667 19040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:05:04.459141 19054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:04.460587 19040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:04.460743 19040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "fa058c83055d4074bd6fbaa45b566087"
format_stamp: "Formatted at 2026-06-19 13:05:04 on dist-test-slave-tpsm"
server_key: "e82c4166b6d99d559b27a0526c135b7f"
server_key_iv: "a651c4f8ff679a68af00fb32c7c457b9"
server_key_version: "encryptionkey@0"
I20260619 13:05:04.460918 19040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:04.479578 19040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:04.480415 19040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:04.480659 19040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:04.499927 19040 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:44887
I20260619 13:05:04.499949 19105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:44887 every 8 connection(s)
I20260619 13:05:04.501497 19040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:04.505021 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 19040
I20260619 13:05:04.505206 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:04.505656 19106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:04.505776 1373 external_mini_cluster.cc:1428] Setting key c2066b4c9cf3b77fb10d8a7846397155
I20260619 13:05:04.520349 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:04.507919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.1.87.126:44887)
0619 13:05:04.508918 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:04.508931 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:04.508976 (+ 45us) server_negotiation.cc:407] Connection header received
0619 13:05:04.510755 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:04.510777 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:04.511225 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:04.511641 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:04.512291 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:04.513605 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:04.515715 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:04.516392 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:04.516522 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:04.517953 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:04.517961 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:04.517975 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:04.518583 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:04.518673 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:04.518694 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:04.518793 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:04.519047 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 13:05:04.519406 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":371,"threads_started":1}
I20260619 13:05:04.521097 19106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087: Bootstrap starting.
I20260619 13:05:04.524674 19106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:04.525717 19106 log.cc:824] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:04.528983 19106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087: No bootstrap required, opened a new log
I20260619 13:05:04.537581 19106 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa058c83055d4074bd6fbaa45b566087" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44887 } }
I20260619 13:05:04.538025 19106 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:04.538141 19106 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa058c83055d4074bd6fbaa45b566087, State: Initialized, Role: FOLLOWER
I20260619 13:05:04.538723 19106 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa058c83055d4074bd6fbaa45b566087" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44887 } }
I20260619 13:05:04.538980 19106 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:04.539131 19106 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:04.539350 19106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:04.541793 19106 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa058c83055d4074bd6fbaa45b566087" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44887 } }
I20260619 13:05:04.542307 19106 leader_election.cc:302] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa058c83055d4074bd6fbaa45b566087; no voters:
I20260619 13:05:04.543025 19106 leader_election.cc:288] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:04.543182 19111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:04.543948 19111 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [term 1 LEADER]: Becoming Leader. State: Replica: fa058c83055d4074bd6fbaa45b566087, State: Running, Role: LEADER
I20260619 13:05:04.544493 19111 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa058c83055d4074bd6fbaa45b566087" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44887 } }
I20260619 13:05:04.545286 19106 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:04.547484 19112 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa058c83055d4074bd6fbaa45b566087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa058c83055d4074bd6fbaa45b566087" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44887 } } }
I20260619 13:05:04.547843 19112 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:04.547421 19113 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa058c83055d4074bd6fbaa45b566087. Latest consensus state: current_term: 1 leader_uuid: "fa058c83055d4074bd6fbaa45b566087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa058c83055d4074bd6fbaa45b566087" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44887 } } }
I20260619 13:05:04.548686 19113 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:04.552186 19119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:04.557682 19119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:04.567027 19119 catalog_manager.cc:1347] Generated new cluster ID: a972181cd8e1426d9e47f5a9758e16e0
I20260619 13:05:04.567173 19119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:04.574970 19119 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:04.576320 19119 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:04.582927 19119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087: Generated new TSK 0
I20260619 13:05:04.583842 19119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:04.595098 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44887
--builtin_ntp_servers=127.1.87.84:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:05:04.923971 19130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:04.924245 19130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:04.924407 19130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:04.931684 19130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:04.931792 19130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:04.931965 19130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:05:04.941572 19130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:04.942885 19130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:04.944625 19130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:04.959139 19135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:04.959193 19136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:04.959219 19138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:04.960071 19130 server_base.cc:1034] running on GCE node
I20260619 13:05:04.960662 19130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:04.961602 19130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:04.962843 19130 hybrid_clock.cc:648] HybridClock initialized: now 1781874304962803 us; error 62 us; skew 500 ppm
I20260619 13:05:04.963172 19130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:04.965723 19130 webserver.cc:460] Webserver started at http://127.1.87.65:42329/ using document root <none> and password file <none>
I20260619 13:05:04.966435 19130 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:04.966573 19130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:04.966888 19130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:04.969715 19130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "0016d3a47d0644059ebc47a541f1b27c"
format_stamp: "Formatted at 2026-06-19 13:05:04 on dist-test-slave-tpsm"
server_key: "a1f66806bd4915bd1072d7375507b587"
server_key_iv: "06fa64bccb1a06bc88b399b8cdc324be"
server_key_version: "encryptionkey@0"
I20260619 13:05:04.970523 19130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "0016d3a47d0644059ebc47a541f1b27c"
format_stamp: "Formatted at 2026-06-19 13:05:04 on dist-test-slave-tpsm"
server_key: "a1f66806bd4915bd1072d7375507b587"
server_key_iv: "06fa64bccb1a06bc88b399b8cdc324be"
server_key_version: "encryptionkey@0"
I20260619 13:05:04.975538 19130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:05:04.978955 19144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:04.980334 19130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:04.980521 19130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "0016d3a47d0644059ebc47a541f1b27c"
format_stamp: "Formatted at 2026-06-19 13:05:04 on dist-test-slave-tpsm"
server_key: "a1f66806bd4915bd1072d7375507b587"
server_key_iv: "06fa64bccb1a06bc88b399b8cdc324be"
server_key_version: "encryptionkey@0"
I20260619 13:05:04.980764 19130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:05.013607 19130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:05.014479 19130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:05.014772 19130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:05.015794 19130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:05.017302 19130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:05.017443 19130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:05.017577 19130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:05.017666 19130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:05.051283 19130 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:41761
I20260619 13:05:05.051409 19256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:41761 every 8 connection(s)
I20260619 13:05:05.052829 19130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:05:05.059976 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 19130
I20260619 13:05:05.060169 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:05:05.060618 1373 external_mini_cluster.cc:1428] Setting key 8bdc422c97633f973a58fd1d7f2d9fad
I20260619 13:05:05.063221 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44887
--builtin_ntp_servers=127.1.87.84:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:05:05.065446 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:05.055429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:55041 (local address 127.1.87.126:44887)
0619 13:05:05.055866 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:05.055871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:05.057009 (+ 1138us) server_negotiation.cc:407] Connection header received
0619 13:05:05.058917 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:05.058921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:05.058994 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:05.059106 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:05.060180 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.060967 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.062524 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.062939 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.063074 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:05.064583 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:05.064588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:05.064591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:05.064614 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:05.064651 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:05.064663 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:05.064760 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:05.064990 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 13:05:05.065083 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":224,"threads_started":1}
I20260619 13:05:05.066404 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:05.055992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44887 (local address 127.1.87.65:55041)
0619 13:05:05.056858 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:05.056892 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:05.058647 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:05.059303 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:05.059371 (+ 68us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:05.059472 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:05.060005 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:05.060021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.061131 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.061135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:05.062365 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:05.062378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.063165 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.063175 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:05.063476 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:05.064196 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:05:05.064199 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:05.064278 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:05:05.064375 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:05.064760 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:05.064774 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:05.064783 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 13:05:05.064963 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 13:05:05.065348 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":283,"threads_started":1}
I20260619 13:05:05.067947 19257 heartbeater.cc:347] Connected to a master server at 127.1.87.126:44887
I20260619 13:05:05.068254 19257 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:05.069132 19257 heartbeater.cc:511] Master 127.1.87.126:44887 requested a full tablet report, sending...
I20260619 13:05:05.071256 19071 ts_manager.cc:194] Registered new tserver with Master: 0016d3a47d0644059ebc47a541f1b27c (127.1.87.65:41761)
I20260619 13:05:05.072762 19071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.65:55041
W20260619 13:05:05.393725 19261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:05.394018 19261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:05.394119 19261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:05.401530 19261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:05.401635 19261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:05.401743 19261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:05:05.410887 19261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:05.412178 19261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:05.414413 19261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:05.427199 19269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:05.427276 19267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:05.427237 19266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:05.428453 19261 server_base.cc:1034] running on GCE node
I20260619 13:05:05.429167 19261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:05.430207 19261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:05.431421 19261 hybrid_clock.cc:648] HybridClock initialized: now 1781874305431326 us; error 75 us; skew 500 ppm
I20260619 13:05:05.431766 19261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:05.434221 19261 webserver.cc:460] Webserver started at http://127.1.87.66:33839/ using document root <none> and password file <none>
I20260619 13:05:05.434926 19261 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:05.435062 19261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:05.435460 19261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:05.439731 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "e7a8281c8336494ca39e320c5b950389"
format_stamp: "Formatted at 2026-06-19 13:05:05 on dist-test-slave-tpsm"
server_key: "e7fb9bd917742b6b958d0796cf8811dd"
server_key_iv: "05a30aad05c1f03b41ac280405bde017"
server_key_version: "encryptionkey@0"
I20260619 13:05:05.440788 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "e7a8281c8336494ca39e320c5b950389"
format_stamp: "Formatted at 2026-06-19 13:05:05 on dist-test-slave-tpsm"
server_key: "e7fb9bd917742b6b958d0796cf8811dd"
server_key_iv: "05a30aad05c1f03b41ac280405bde017"
server_key_version: "encryptionkey@0"
I20260619 13:05:05.446761 19261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 13:05:05.450135 19275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:05.451627 19261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:05.451779 19261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "e7a8281c8336494ca39e320c5b950389"
format_stamp: "Formatted at 2026-06-19 13:05:05 on dist-test-slave-tpsm"
server_key: "e7fb9bd917742b6b958d0796cf8811dd"
server_key_iv: "05a30aad05c1f03b41ac280405bde017"
server_key_version: "encryptionkey@0"
I20260619 13:05:05.451959 19261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:05.474020 19261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:05.474893 19261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:05.475183 19261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:05.476217 19261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:05.477728 19261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:05.477811 19261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:05.477958 19261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:05.478058 19261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:05.512095 19261 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:38201
I20260619 13:05:05.512142 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:38201 every 8 connection(s)
I20260619 13:05:05.513302 19261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:05:05.520995 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 19261
I20260619 13:05:05.521178 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:05:05.521588 1373 external_mini_cluster.cc:1428] Setting key cdd1b1f33d5e0141bfa72dbce5a23bf7
I20260619 13:05:05.524518 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44887
--builtin_ntp_servers=127.1.87.84:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:05:05.526466 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:05.516215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:37923 (local address 127.1.87.126:44887)
0619 13:05:05.516386 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:05.516391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:05.517837 (+ 1446us) server_negotiation.cc:407] Connection header received
0619 13:05:05.519692 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:05.519699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:05.519794 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:05.519907 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:05.520924 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.521833 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.523635 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.524142 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.524278 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:05.525687 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:05.525693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:05.525697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:05.525722 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:05.525751 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:05.525763 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:05.525855 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:05.526065 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 13:05:05.526166 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:05:05.527472 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:05.516793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44887 (local address 127.1.87.66:37923)
0619 13:05:05.517680 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:05.517719 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:05.519432 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:05.520122 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:05.520129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:05.520219 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:05.520752 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:05.520767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.521988 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.521993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:05.523495 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:05.523506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.524356 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.524363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:05.524592 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:05.525381 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:05:05.525384 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:05.525421 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:05:05.525495 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:05.525855 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:05.525866 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:05.525876 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:05:05.526054 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 13:05:05.526420 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":286,"threads_started":1}
I20260619 13:05:05.528790 19388 heartbeater.cc:347] Connected to a master server at 127.1.87.126:44887
I20260619 13:05:05.529140 19388 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:05.530092 19388 heartbeater.cc:511] Master 127.1.87.126:44887 requested a full tablet report, sending...
I20260619 13:05:05.531733 19071 ts_manager.cc:194] Registered new tserver with Master: e7a8281c8336494ca39e320c5b950389 (127.1.87.66:38201)
I20260619 13:05:05.532778 19071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.66:37923
W20260619 13:05:05.859069 19391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:05.859469 19391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:05.859629 19391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:05.867216 19391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:05.867450 19391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:05.867677 19391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:05:05.877156 19391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:44887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:05.878772 19391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:05.881387 19391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:05.896833 19396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:05.896801 19397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:05.896760 19399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:05.898113 19391 server_base.cc:1034] running on GCE node
I20260619 13:05:05.898957 19391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:05.900048 19391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:05.901271 19391 hybrid_clock.cc:648] HybridClock initialized: now 1781874305901206 us; error 54 us; skew 500 ppm
I20260619 13:05:05.901646 19391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:05.904126 19391 webserver.cc:460] Webserver started at http://127.1.87.67:45069/ using document root <none> and password file <none>
I20260619 13:05:05.904911 19391 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:05.905045 19391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:05.905387 19391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:05.908574 19391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "8823624503e64cd4b79ec163252cce20"
format_stamp: "Formatted at 2026-06-19 13:05:05 on dist-test-slave-tpsm"
server_key: "1c66a8e6d46c9a8fd6f2368e0d63b471"
server_key_iv: "441f71542a2e7accdffbd00619867b00"
server_key_version: "encryptionkey@0"
I20260619 13:05:05.909430 19391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "8823624503e64cd4b79ec163252cce20"
format_stamp: "Formatted at 2026-06-19 13:05:05 on dist-test-slave-tpsm"
server_key: "1c66a8e6d46c9a8fd6f2368e0d63b471"
server_key_iv: "441f71542a2e7accdffbd00619867b00"
server_key_version: "encryptionkey@0"
I20260619 13:05:05.915061 19391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:05:05.918773 19405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:05.920360 19391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:05.920562 19391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "8823624503e64cd4b79ec163252cce20"
format_stamp: "Formatted at 2026-06-19 13:05:05 on dist-test-slave-tpsm"
server_key: "1c66a8e6d46c9a8fd6f2368e0d63b471"
server_key_iv: "441f71542a2e7accdffbd00619867b00"
server_key_version: "encryptionkey@0"
I20260619 13:05:05.920814 19391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:05.941711 19391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:05.942610 19391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:05.942902 19391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:05.943977 19391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:05.945607 19391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:05.945734 19391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:05.945874 19391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:05.945974 19391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:05.982681 19391 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:41779
I20260619 13:05:05.982738 19517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:41779 every 8 connection(s)
I20260619 13:05:05.984126 19391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:05:05.991717 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 19391
I20260619 13:05:05.991912 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:05:05.992537 1373 external_mini_cluster.cc:1428] Setting key 364c82ccfe46b0a5fcd81ca427499e5b
I20260619 13:05:05.997328 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:05.986752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:52083 (local address 127.1.87.126:44887)
0619 13:05:05.986932 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:05.986937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:05.988440 (+ 1503us) server_negotiation.cc:407] Connection header received
0619 13:05:05.990099 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:05.990104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:05.990186 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:05.990280 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:05.991610 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.992752 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.994686 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.995171 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.995347 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:05.996637 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:05.996642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:05.996646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:05.996664 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:05.996695 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:05.996707 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:05.996787 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:05.996984 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 13:05:05.997062 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:05:05.998277 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:05.987236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44887 (local address 127.1.87.67:52083)
0619 13:05:05.988237 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:05.988276 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:05.989832 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:05.990596 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:05.990604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:05.990690 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:05.991277 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:05.991294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.992909 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.992914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:05.994530 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:05.994550 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:05.995464 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:05.995473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:05.995709 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:05.996330 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:05:05.996332 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:05.996372 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:05:05.996452 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:05.996798 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:05.996805 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:05.996811 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:05:05.996944 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 13:05:05.997242 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":278,"threads_started":1}
I20260619 13:05:05.999782 19518 heartbeater.cc:347] Connected to a master server at 127.1.87.126:44887
I20260619 13:05:06.000079 19518 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:06.000803 19518 heartbeater.cc:511] Master 127.1.87.126:44887 requested a full tablet report, sending...
I20260619 13:05:06.002189 19071 ts_manager.cc:194] Registered new tserver with Master: 8823624503e64cd4b79ec163252cce20 (127.1.87.67:41779)
I20260619 13:05:06.002977 19071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.67:52083
I20260619 13:05:06.008250 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:05:06.019016 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.012554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.1.87.126:44887)
0619 13:05:06.012832 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.012839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.013155 (+ 316us) server_negotiation.cc:407] Connection header received
0619 13:05:06.013296 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.013300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.013375 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.013462 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:06.014073 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.015002 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.016574 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.017060 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.017176 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.018174 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:06.018180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:06.018183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:06.018204 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:06.018236 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:06.018248 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:06.018326 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.018471 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.018679 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:05:06.023115 19071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:05:06.030814 19071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:05:06.072573 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.044328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.1.87.67:41779)
0619 13:05:06.055983 (+ 11655us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.055997 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.062579 (+ 6582us) server_negotiation.cc:407] Connection header received
0619 13:05:06.062792 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.062799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.063021 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.063210 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:06.065019 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.066842 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.069963 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.071405 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.071557 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.071854 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.072059 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.072221 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11400,"thread_start_us":19843,"threads_started":1}
I20260619 13:05:06.072603 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.044050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:41779 (local address 127.0.0.1:55596)
0619 13:05:06.062454 (+ 18404us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:06.062480 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:06.062639 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:06.064240 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:06.064248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:06.064284 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:06.064753 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.064768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.067003 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.067008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.069661 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.069675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.071594 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.071600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.071785 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.071842 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 13:05:06.071997 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 13:05:06.072191 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18198,"mutex_wait_us":3,"thread_start_us":18685,"threads_started":1}
I20260619 13:05:06.073735 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.043817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:41761 (local address 127.0.0.1:37108)
0619 13:05:06.052935 (+ 9118us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:06.052983 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:06.053184 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:06.063362 (+ 10178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:06.063369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:06.063402 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:06.063995 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.064014 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.067898 (+ 3884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.067903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.070625 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.070641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.072853 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.072859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.073049 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.073086 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:05:06.073190 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 13:05:06.073339 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8826,"mutex_wait_us":133,"thread_start_us":22066,"threads_started":1}
I20260619 13:05:06.073771 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.044032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.1.87.65:41761)
0619 13:05:06.060845 (+ 16813us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.060868 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.060911 (+ 43us) server_negotiation.cc:407] Connection header received
0619 13:05:06.061055 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.061069 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.061454 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.063036 (+ 1582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:06.066235 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.067757 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.070850 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.072296 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.072433 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.072710 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.073264 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.073421 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16575,"thread_start_us":16400,"threads_started":1}
I20260619 13:05:06.074808 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.046484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:38201 (local address 127.0.0.1:41796)
0619 13:05:06.066167 (+ 19683us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:06.066188 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:06.066307 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:06.066873 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:06.066877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:06.066901 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:06.067535 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.067551 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.069451 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.069456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.071378 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.071392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.073638 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.073646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.074267 (+ 621us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.074291 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:05:06.074402 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 13:05:06.074531 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19547,"thread_start_us":6006,"threads_started":1}
I20260619 13:05:06.074959 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.047144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.1.87.66:38201)
0619 13:05:06.066208 (+ 19064us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.066219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.066252 (+ 33us) server_negotiation.cc:407] Connection header received
0619 13:05:06.066494 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.066500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.066684 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.066885 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:06.067700 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.069295 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.071590 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.073164 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.073287 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.073507 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.074473 (+ 966us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.074643 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18878,"thread_start_us":19093,"threads_started":1}
I20260619 13:05:06.076850 19453 tablet_service.cc:1467] Processing CreateTablet for tablet 30a216c5db5e455f83308d1214b563e7 (DEFAULT_TABLE table=test-table [id=bb3ec1b7a3ff49758bb42962aa85b4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:06.078104 19453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a216c5db5e455f83308d1214b563e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:06.078570 19192 tablet_service.cc:1467] Processing CreateTablet for tablet 30a216c5db5e455f83308d1214b563e7 (DEFAULT_TABLE table=test-table [id=bb3ec1b7a3ff49758bb42962aa85b4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:06.078570 19323 tablet_service.cc:1467] Processing CreateTablet for tablet 30a216c5db5e455f83308d1214b563e7 (DEFAULT_TABLE table=test-table [id=bb3ec1b7a3ff49758bb42962aa85b4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:06.079478 19257 heartbeater.cc:503] Master 127.1.87.126:44887 was elected leader, sending a full tablet report...
I20260619 13:05:06.079953 19192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a216c5db5e455f83308d1214b563e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:06.080137 19323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a216c5db5e455f83308d1214b563e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:06.092962 19536 tablet_bootstrap.cc:492] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: Bootstrap starting.
I20260619 13:05:06.095069 19538 tablet_bootstrap.cc:492] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: Bootstrap starting.
I20260619 13:05:06.096781 19537 tablet_bootstrap.cc:492] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: Bootstrap starting.
I20260619 13:05:06.097154 19536 tablet_bootstrap.cc:654] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:06.098537 19536 log.cc:824] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:06.098634 19538 tablet_bootstrap.cc:654] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:06.099749 19538 log.cc:824] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:06.099820 19537 tablet_bootstrap.cc:654] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:06.100989 19537 log.cc:824] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:06.102173 19536 tablet_bootstrap.cc:492] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: No bootstrap required, opened a new log
I20260619 13:05:06.102705 19538 tablet_bootstrap.cc:492] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: No bootstrap required, opened a new log
I20260619 13:05:06.102880 19536 ts_tablet_manager.cc:1397] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 13:05:06.102967 19538 ts_tablet_manager.cc:1397] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:05:06.104606 19537 tablet_bootstrap.cc:492] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: No bootstrap required, opened a new log
I20260619 13:05:06.105082 19537 ts_tablet_manager.cc:1397] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:05:06.112654 19537 raft_consensus.cc:348] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } }
I20260619 13:05:06.113092 19537 raft_consensus.cc:374] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:06.113196 19537 raft_consensus.cc:729] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0016d3a47d0644059ebc47a541f1b27c, State: Initialized, Role: FOLLOWER
I20260619 13:05:06.113860 19537 consensus_queue.cc:260] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } }
I20260619 13:05:06.113862 19536 raft_consensus.cc:348] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } }
I20260619 13:05:06.113862 19538 raft_consensus.cc:348] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } }
I20260619 13:05:06.114301 19538 raft_consensus.cc:374] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:06.114301 19536 raft_consensus.cc:374] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:06.114408 19538 raft_consensus.cc:729] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a8281c8336494ca39e320c5b950389, State: Initialized, Role: FOLLOWER
I20260619 13:05:06.114406 19536 raft_consensus.cc:729] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8823624503e64cd4b79ec163252cce20, State: Initialized, Role: FOLLOWER
I20260619 13:05:06.115093 19536 consensus_queue.cc:260] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } }
I20260619 13:05:06.115087 19538 consensus_queue.cc:260] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } }
I20260619 13:05:06.115857 19537 ts_tablet_manager.cc:1428] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 13:05:06.118646 19518 heartbeater.cc:503] Master 127.1.87.126:44887 was elected leader, sending a full tablet report...
I20260619 13:05:06.118692 19388 heartbeater.cc:503] Master 127.1.87.126:44887 was elected leader, sending a full tablet report...
I20260619 13:05:06.118901 19538 ts_tablet_manager.cc:1428] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
I20260619 13:05:06.120433 19536 ts_tablet_manager.cc:1428] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260619 13:05:06.143445 19544 raft_consensus.cc:482] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:05:06.143844 19544 raft_consensus.cc:504] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } }
I20260619 13:05:06.145766 19544 leader_election.cc:288] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0016d3a47d0644059ebc47a541f1b27c (127.1.87.65:41761), 8823624503e64cd4b79ec163252cce20 (127.1.87.67:41779)
I20260619 13:05:06.154974 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.146328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:55333 (local address 127.1.87.65:41761)
0619 13:05:06.146687 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.146696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.147203 (+ 507us) server_negotiation.cc:407] Connection header received
0619 13:05:06.147440 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.147445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.147540 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.147646 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:06.148393 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.149533 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.152747 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.153974 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.154116 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.154229 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.154589 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.154729 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260619 13:05:06.155066 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.146259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:58351 (local address 127.1.87.67:41779)
0619 13:05:06.146478 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.146487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.147437 (+ 950us) server_negotiation.cc:407] Connection header received
0619 13:05:06.147611 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.147618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.147706 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.147798 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:06.148520 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.149567 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.152699 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.154005 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.154123 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.154227 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.154703 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.154836 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:05:06.155092 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.145982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:41761 (local address 127.1.87.66:55333)
0619 13:05:06.147045 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:06.147073 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:06.147259 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:06.147678 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:06.147681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:06.147703 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:06.148227 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.148246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.149718 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.149724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.152427 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.152443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.154189 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.154197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.154412 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.154455 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 13:05:06.154530 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 13:05:06.154666 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":148,"thread_start_us":633,"threads_started":1}
I20260619 13:05:06.155139 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.146242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:41779 (local address 127.1.87.66:58351)
0619 13:05:06.147349 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:06.147369 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:06.147490 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:06.147835 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:06.147839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:06.147883 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:06.148350 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.148364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.149719 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.149724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.152427 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:06.152443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.154358 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.154363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:06.154491 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.154549 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 13:05:06.154646 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 13:05:06.154764 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":191,"thread_start_us":720,"threads_started":1}
I20260619 13:05:06.156246 19473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a216c5db5e455f83308d1214b563e7" candidate_uuid: "e7a8281c8336494ca39e320c5b950389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8823624503e64cd4b79ec163252cce20" is_pre_election: true
I20260619 13:05:06.156471 19212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a216c5db5e455f83308d1214b563e7" candidate_uuid: "e7a8281c8336494ca39e320c5b950389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0016d3a47d0644059ebc47a541f1b27c" is_pre_election: true
I20260619 13:05:06.156931 19473 raft_consensus.cc:2454] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a8281c8336494ca39e320c5b950389 in term 0.
I20260619 13:05:06.156953 19212 raft_consensus.cc:2454] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a8281c8336494ca39e320c5b950389 in term 0.
I20260619 13:05:06.157647 19276 leader_election.cc:302] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0016d3a47d0644059ebc47a541f1b27c, e7a8281c8336494ca39e320c5b950389; no voters:
I20260619 13:05:06.158025 19544 raft_consensus.cc:2781] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:05:06.158149 19544 raft_consensus.cc:482] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:05:06.158293 19544 raft_consensus.cc:3037] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:06.160559 19544 raft_consensus.cc:504] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } }
I20260619 13:05:06.161230 19544 leader_election.cc:288] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [CANDIDATE]: Term 1 election: Requested vote from peers 0016d3a47d0644059ebc47a541f1b27c (127.1.87.65:41761), 8823624503e64cd4b79ec163252cce20 (127.1.87.67:41779)
I20260619 13:05:06.161727 19473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a216c5db5e455f83308d1214b563e7" candidate_uuid: "e7a8281c8336494ca39e320c5b950389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8823624503e64cd4b79ec163252cce20"
I20260619 13:05:06.161727 19212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a216c5db5e455f83308d1214b563e7" candidate_uuid: "e7a8281c8336494ca39e320c5b950389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0016d3a47d0644059ebc47a541f1b27c"
I20260619 13:05:06.161959 19212 raft_consensus.cc:3037] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:06.161948 19473 raft_consensus.cc:3037] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:06.165262 19212 raft_consensus.cc:2454] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a8281c8336494ca39e320c5b950389 in term 1.
I20260619 13:05:06.165262 19473 raft_consensus.cc:2454] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a8281c8336494ca39e320c5b950389 in term 1.
I20260619 13:05:06.165832 19276 leader_election.cc:302] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0016d3a47d0644059ebc47a541f1b27c, e7a8281c8336494ca39e320c5b950389; no voters:
I20260619 13:05:06.166138 19544 raft_consensus.cc:2781] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:06.166605 19544 raft_consensus.cc:686] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 1 LEADER]: Becoming Leader. State: Replica: e7a8281c8336494ca39e320c5b950389, State: Running, Role: LEADER
I20260619 13:05:06.167094 19544 consensus_queue.cc:237] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } }
I20260619 13:05:06.171555 19071 catalog_manager.cc:5495] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7a8281c8336494ca39e320c5b950389 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "e7a8281c8336494ca39e320c5b950389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7a8281c8336494ca39e320c5b950389" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 38201 } health_report { overall_health: HEALTHY } } }
I20260619 13:05:06.187987 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.182021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.1.87.66:38201)
0619 13:05:06.182234 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.182241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.182312 (+ 71us) server_negotiation.cc:407] Connection header received
0619 13:05:06.182594 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.182598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.182696 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.182847 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:05:06.183226 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.184259 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.185820 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.186240 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.186378 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.186861 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:05:06.187415 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:05:06.187583 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.187634 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.187713 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:05:06.196168 19473 raft_consensus.cc:1261] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [term 1 FOLLOWER]: Refusing update from remote peer e7a8281c8336494ca39e320c5b950389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:06.196168 19212 raft_consensus.cc:1261] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [term 1 FOLLOWER]: Refusing update from remote peer e7a8281c8336494ca39e320c5b950389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:06.197443 19547 consensus_queue.cc:1035] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8823624503e64cd4b79ec163252cce20" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:06.197806 19544 consensus_queue.cc:1035] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0016d3a47d0644059ebc47a541f1b27c" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:06.211556 19552 mvcc.cc:205] Tried to move back new op lower bound from 7298557158168985600 to 7298557158064254976. Current Snapshot: MvccSnapshot[applied={T|T < 7298557158168985600}]
I20260619 13:05:06.212456 19550 mvcc.cc:205] Tried to move back new op lower bound from 7298557158168985600 to 7298557158064254976. Current Snapshot: MvccSnapshot[applied={T|T < 7298557158168985600}]
I20260619 13:05:06.230695 19071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:06.230979 19071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:06.235003 19071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa058c83055d4074bd6fbaa45b566087: Sending DeleteTablet for 3 replicas of tablet 30a216c5db5e455f83308d1214b563e7
I20260619 13:05:06.236634 19192 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a216c5db5e455f83308d1214b563e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:06 UTC) from {username='slave'} at 127.0.0.1:37108
I20260619 13:05:06.236709 19453 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a216c5db5e455f83308d1214b563e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:06 UTC) from {username='slave'} at 127.0.0.1:55596
I20260619 13:05:06.237995 19560 tablet_replica.cc:330] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: stopping tablet replica
I20260619 13:05:06.238013 19323 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a216c5db5e455f83308d1214b563e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:06 UTC) from {username='slave'} at 127.0.0.1:41796
I20260619 13:05:06.238533 19560 raft_consensus.cc:2229] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:06.239091 19560 raft_consensus.cc:2258] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:06.239961 19561 tablet_replica.cc:330] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: stopping tablet replica
I20260619 13:05:06.240468 19561 raft_consensus.cc:2229] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:06.240964 19561 raft_consensus.cc:2258] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:06.241928 19560 ts_tablet_manager.cc:1905] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:06.243592 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 19130
I20260619 13:05:06.249459 19560 ts_tablet_manager.cc:1918] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:06.249701 19560 log.cc:1196] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/30a216c5db5e455f83308d1214b563e7
I20260619 13:05:06.250247 19560 ts_tablet_manager.cc:1939] T 30a216c5db5e455f83308d1214b563e7 P 0016d3a47d0644059ebc47a541f1b27c: Deleting consensus metadata
I20260619 13:05:06.253438 19561 ts_tablet_manager.cc:1905] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:06.261484 19561 ts_tablet_manager.cc:1918] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:06.261713 19561 log.cc:1196] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/30a216c5db5e455f83308d1214b563e7
I20260619 13:05:06.262253 19561 ts_tablet_manager.cc:1939] T 30a216c5db5e455f83308d1214b563e7 P 8823624503e64cd4b79ec163252cce20: Deleting consensus metadata
I20260619 13:05:06.263863 19562 tablet_replica.cc:330] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: stopping tablet replica
I20260619 13:05:06.264297 19057 catalog_manager.cc:4841] TS 8823624503e64cd4b79ec163252cce20 (127.1.87.67:41779): tablet 30a216c5db5e455f83308d1214b563e7 (table test-table [id=bb3ec1b7a3ff49758bb42962aa85b4ac]) successfully deleted
I20260619 13:05:06.264464 19562 raft_consensus.cc:2229] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:05:06.265053 19055 catalog_manager.cc:4841] TS 0016d3a47d0644059ebc47a541f1b27c (127.1.87.65:41761): tablet 30a216c5db5e455f83308d1214b563e7 (table test-table [id=bb3ec1b7a3ff49758bb42962aa85b4ac]) successfully deleted
I20260619 13:05:06.265395 19562 raft_consensus.cc:2258] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:06.267895 19562 ts_tablet_manager.cc:1905] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:06.269508 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.245216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.1.87.65:41761)
0619 13:05:06.245460 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.245470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.256788 (+ 11318us) server_negotiation.cc:407] Connection header received
0619 13:05:06.257117 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.257126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.257287 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.257639 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:06.258677 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.260522 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.265082 (+ 4560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.265745 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.265865 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.267828 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:06.267838 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:06.267855 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:06.268614 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:06.268685 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:06.268711 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:06.268835 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.268991 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.269114 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:05:06.270263 19252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:06.276871 19562 ts_tablet_manager.cc:1918] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:06.277236 19562 log.cc:1196] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/30a216c5db5e455f83308d1214b563e7
I20260619 13:05:06.278010 19562 ts_tablet_manager.cc:1939] T 30a216c5db5e455f83308d1214b563e7 P e7a8281c8336494ca39e320c5b950389: Deleting consensus metadata
I20260619 13:05:06.280117 19056 catalog_manager.cc:4841] TS e7a8281c8336494ca39e320c5b950389 (127.1.87.66:38201): tablet 30a216c5db5e455f83308d1214b563e7 (table test-table [id=bb3ec1b7a3ff49758bb42962aa85b4ac]) successfully deleted
I20260619 13:05:06.365059 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 19130
I20260619 13:05:06.383142 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 19261
I20260619 13:05:06.392277 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.384274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.1.87.66:38201)
0619 13:05:06.384492 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.384499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.384708 (+ 209us) server_negotiation.cc:407] Connection header received
0619 13:05:06.384871 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.384878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.384999 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.385116 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:06.385799 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.386996 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.388585 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.389267 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.389412 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.390510 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:06.390522 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:06.390547 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:06.391399 (+ 852us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:06.391465 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:06.391490 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:06.391604 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.391782 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.391888 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:05:06.393154 19383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:06.509238 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 19261
I20260619 13:05:06.529196 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 19391
I20260619 13:05:06.536190 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:06.530003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.1.87.67:41779)
0619 13:05:06.530192 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:06.530196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:06.530215 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:05:06.530371 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:06.530375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:06.530453 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:06.530542 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:06.530961 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.531841 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.533133 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:06.533543 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:06.533643 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:06.534789 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:06.534798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:06.534813 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:06.535470 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:06.535518 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:06.535538 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:06.535630 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:06.535755 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 13:05:06.535841 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 13:05:06.536890 19513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:06.627772 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 19391
I20260619 13:05:06.644924 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 19040
I20260619 13:05:06.646226 19101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:06.759433 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 19040
2026-06-19T13:05:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2749 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:06 dist-test-slave-tpsm krb5kdc[19570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:06 dist-test-slave-tpsm krb5kdc[19570](info): set up 2 sockets
Jun 19 13:05:06 dist-test-slave-tpsm krb5kdc[19570](info): commencing operation
krb5kdc: starting...
W20260619 13:05:07.867489 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:08 dist-test-slave-tpsm krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:08.118574 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:45025
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:08.444758 19583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:08.445065 19583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:08.445206 19583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:08.452541 19583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:08.452658 19583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:08.452749 19583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:08.452852 19583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:08.452934 19583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:08.462966 19583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:08.464376 19583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:08.465718 19583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:08.479974 19588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:08.479905 19591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:08.479876 19589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:08.481141 19583 server_base.cc:1034] running on GCE node
I20260619 13:05:08.481995 19583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:08.483590 19583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:08.484838 19583 hybrid_clock.cc:648] HybridClock initialized: now 1781874308484799 us; error 52 us; skew 500 ppm
I20260619 13:05:08.485224 19583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:08 dist-test-slave-tpsm krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874308, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:08.488489 19583 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:08.489969 19583 webserver.cc:460] Webserver started at http://127.1.87.126:39813/ using document root <none> and password file <none>
I20260619 13:05:08.490809 19583 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:08.490921 19583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:08.491173 19583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:08.494362 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42"
format_stamp: "Formatted at 2026-06-19 13:05:08 on dist-test-slave-tpsm"
server_key: "d541d3ba868b4442e3a2df4af28ab225"
server_key_iv: "f561615283ddc62177d819db176564a8"
server_key_version: "encryptionkey@0"
I20260619 13:05:08.495087 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42"
format_stamp: "Formatted at 2026-06-19 13:05:08 on dist-test-slave-tpsm"
server_key: "d541d3ba868b4442e3a2df4af28ab225"
server_key_iv: "f561615283ddc62177d819db176564a8"
server_key_version: "encryptionkey@0"
I20260619 13:05:08.500381 19583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 13:05:08.503866 19598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:08.505308 19583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:05:08.505466 19583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42"
format_stamp: "Formatted at 2026-06-19 13:05:08 on dist-test-slave-tpsm"
server_key: "d541d3ba868b4442e3a2df4af28ab225"
server_key_iv: "f561615283ddc62177d819db176564a8"
server_key_version: "encryptionkey@0"
I20260619 13:05:08.505645 19583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:08.527017 19583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:08.531677 19583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:08.531940 19583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:08.551074 19583 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:45025
I20260619 13:05:08.551110 19649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:45025 every 8 connection(s)
I20260619 13:05:08.552605 19583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:08.556510 19650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:08.562532 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 19583
I20260619 13:05:08.562772 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:08.563201 1373 external_mini_cluster.cc:1428] Setting key ff6bf990aca16e68c988f560d8a0980f
I20260619 13:05:08.567499 19650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42: Bootstrap starting.
I20260619 13:05:08.571282 19650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:08.572536 19650 log.cc:824] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:08.575799 19650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42: No bootstrap required, opened a new log
Jun 19 13:05:08 dist-test-slave-tpsm krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:08.585678 19650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45025 } }
I20260619 13:05:08.586206 19650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:08.586349 19650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9ee8b7672b4ce49e35ba1e26e86d42, State: Initialized, Role: FOLLOWER
I20260619 13:05:08.587074 19650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45025 } }
I20260619 13:05:08.587363 19650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:08.587538 19650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:08.587729 19650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:08.588052 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:08.565109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.1.87.126:45025)
0619 13:05:08.565906 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:08.565917 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:08.565957 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:05:08.567797 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:08.567819 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:08.568205 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:08.568521 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:08.570487 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:08.571996 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:08.573756 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:08.574436 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:08.574566 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:08.580028 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:08.580043 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:08.580064 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:08.580120 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:08.583538 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:08.584378 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:08.584384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:08.584390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:08.584557 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:08.584960 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:08.584963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:08.584964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:08.585491 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:08.585704 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:08.586249 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 13:05:08.586761 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":385,"threads_started":1}
I20260619 13:05:08.590248 19650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45025 } }
I20260619 13:05:08.590780 19650 leader_election.cc:302] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b9ee8b7672b4ce49e35ba1e26e86d42; no voters:
I20260619 13:05:08.591449 19650 leader_election.cc:288] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:08.591595 19655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:08.592345 19655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9ee8b7672b4ce49e35ba1e26e86d42, State: Running, Role: LEADER
I20260619 13:05:08.592948 19655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45025 } }
I20260619 13:05:08.593508 19650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:08.602480 19657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b9ee8b7672b4ce49e35ba1e26e86d42. Latest consensus state: current_term: 1 leader_uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45025 } } }
I20260619 13:05:08.602978 19657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:08.602452 19656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ee8b7672b4ce49e35ba1e26e86d42" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 45025 } } }
I20260619 13:05:08.603114 19656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:08.610446 19664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:08.615832 19664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:08.624564 19664 catalog_manager.cc:1347] Generated new cluster ID: d190196ea3ca4db891af2ae9e1a30261
I20260619 13:05:08.624712 19664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:08.647644 19664 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:08.649336 19664 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:08.667665 19664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42: Generated new TSK 0
I20260619 13:05:08.668581 19664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:05:08.737197 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45025
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:09.101454 19676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:09.101871 19676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:09.102042 19676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:09.109768 19676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:09.109903 19676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:09.110080 19676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:05:09.120286 19676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:09.121649 19676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:09.122942 19676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:09.136478 19681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:09.136514 19684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:09.136554 19682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:09.137746 19676 server_base.cc:1034] running on GCE node
I20260619 13:05:09.138422 19676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:09.139437 19676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:09.140663 19676 hybrid_clock.cc:648] HybridClock initialized: now 1781874309140596 us; error 53 us; skew 500 ppm
I20260619 13:05:09.140995 19676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:09 dist-test-slave-tpsm krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:09.144279 19676 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:05:09.146010 19676 webserver.cc:460] Webserver started at http://127.1.87.65:34063/ using document root <none> and password file <none>
I20260619 13:05:09.146723 19676 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:09.146827 19676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:09.147217 19676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:09.150537 19676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "074047f5005c4d1d8c04018f60571860"
format_stamp: "Formatted at 2026-06-19 13:05:09 on dist-test-slave-tpsm"
server_key: "e455cb4f346858d2b4b47fcc5f18f2cf"
server_key_iv: "4c5912fff918ce36e1e374d6867716c4"
server_key_version: "encryptionkey@0"
I20260619 13:05:09.151393 19676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "074047f5005c4d1d8c04018f60571860"
format_stamp: "Formatted at 2026-06-19 13:05:09 on dist-test-slave-tpsm"
server_key: "e455cb4f346858d2b4b47fcc5f18f2cf"
server_key_iv: "4c5912fff918ce36e1e374d6867716c4"
server_key_version: "encryptionkey@0"
I20260619 13:05:09.156533 19676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 13:05:09.160362 19691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:09.161850 19676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:09.162030 19676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "074047f5005c4d1d8c04018f60571860"
format_stamp: "Formatted at 2026-06-19 13:05:09 on dist-test-slave-tpsm"
server_key: "e455cb4f346858d2b4b47fcc5f18f2cf"
server_key_iv: "4c5912fff918ce36e1e374d6867716c4"
server_key_version: "encryptionkey@0"
I20260619 13:05:09.162257 19676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:09.178694 19676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:09.183580 19676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:09.183882 19676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:09.184893 19676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:09.186242 19676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:09.186316 19676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:09.186477 19676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:09.186556 19676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:09.223168 19676 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:39163
I20260619 13:05:09.223274 19803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:39163 every 8 connection(s)
I20260619 13:05:09.224517 19676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:05:09.228870 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 19676
I20260619 13:05:09.229071 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:05:09.229645 1373 external_mini_cluster.cc:1428] Setting key ce7fe1651e4272f89e9e55e67532d8e5
Jun 19 13:05:09 dist-test-slave-tpsm krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:09.249758 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:09.227211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:41507 (local address 127.1.87.126:45025)
0619 13:05:09.227755 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:09.227763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:09.228682 (+ 919us) server_negotiation.cc:407] Connection header received
0619 13:05:09.231159 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:09.231164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:09.231238 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:09.231407 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:09.233783 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:09.234640 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:09.237116 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:09.237634 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:09.237771 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:09.242169 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:09.242179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:09.242187 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:09.242233 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:09.245922 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:09.246770 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:09.246778 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:09.246784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:09.246933 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:09.247636 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:09.247644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:09.247650 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:09.248045 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:09.248213 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:09.248953 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 13:05:09.249240 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":292,"threads_started":1}
I20260619 13:05:09.250384 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:09.227703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45025 (local address 127.1.87.65:41507)
0619 13:05:09.228481 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:09.228538 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:09.230861 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:09.231778 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:09.231793 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:09.232684 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:09.233573 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:09.233602 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:09.234815 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:09.234823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:09.236957 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:09.236976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:09.237899 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:09.237908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:09.238257 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:09.239018 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:09.239061 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:09.241930 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:09.246106 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:09.246114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:09.246129 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:09.246618 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:09.247053 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:09.247057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:09.247059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:09.247497 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:09.248196 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:09.248203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:09.248552 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 13:05:09.248786 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 13:05:09.249204 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":286,"threads_started":1}
I20260619 13:05:09.252022 19804 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45025
I20260619 13:05:09.252357 19804 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:09.253273 19804 heartbeater.cc:511] Master 127.1.87.126:45025 requested a full tablet report, sending...
I20260619 13:05:09.255676 19615 ts_manager.cc:194] Registered new tserver with Master: 074047f5005c4d1d8c04018f60571860 (127.1.87.65:39163)
I20260619 13:05:09.257098 19615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:41507
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:05:09.289005 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45025
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:09.629009 19810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:09.629330 19810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:09.629607 19810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:09.637205 19810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:09.637331 19810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:09.637521 19810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:05:09.648398 19810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:09.649883 19810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:09.651201 19810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:09.664429 19816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:09.664389 19818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:09.665028 19815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:09.665315 19810 server_base.cc:1034] running on GCE node
I20260619 13:05:09.665964 19810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:09.666950 19810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:09.668133 19810 hybrid_clock.cc:648] HybridClock initialized: now 1781874309668089 us; error 31 us; skew 500 ppm
I20260619 13:05:09.668463 19810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:09 dist-test-slave-tpsm krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:09.676774 19810 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:05:09.678539 19810 webserver.cc:460] Webserver started at http://127.1.87.66:35501/ using document root <none> and password file <none>
I20260619 13:05:09.679366 19810 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:09.679526 19810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:09.679859 19810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:09.682859 19810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2"
format_stamp: "Formatted at 2026-06-19 13:05:09 on dist-test-slave-tpsm"
server_key: "94909a7a2683cdc87fa6b208d5a0be3e"
server_key_iv: "7927609e7dce76a741b63465b5a39ea7"
server_key_version: "encryptionkey@0"
I20260619 13:05:09.683723 19810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2"
format_stamp: "Formatted at 2026-06-19 13:05:09 on dist-test-slave-tpsm"
server_key: "94909a7a2683cdc87fa6b208d5a0be3e"
server_key_iv: "7927609e7dce76a741b63465b5a39ea7"
server_key_version: "encryptionkey@0"
I20260619 13:05:09.688966 19810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 13:05:09.692904 19825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:09.694567 19810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:09.694779 19810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2"
format_stamp: "Formatted at 2026-06-19 13:05:09 on dist-test-slave-tpsm"
server_key: "94909a7a2683cdc87fa6b208d5a0be3e"
server_key_iv: "7927609e7dce76a741b63465b5a39ea7"
server_key_version: "encryptionkey@0"
I20260619 13:05:09.695104 19810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:09.705749 19810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:09.710335 19810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:09.710687 19810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:09.711709 19810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:09.713001 19810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:09.713111 19810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:09.713263 19810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:09.713335 19810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:09.749238 19810 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:35771
I20260619 13:05:09.749310 19937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:35771 every 8 connection(s)
I20260619 13:05:09.750747 19810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:05:09.755090 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 19810
I20260619 13:05:09.755275 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:05:09.755865 1373 external_mini_cluster.cc:1428] Setting key bebab0500ca9e7e2558c9822ff8a9414
Jun 19 13:05:09 dist-test-slave-tpsm krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:09.774689 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:09.753407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:36185 (local address 127.1.87.126:45025)
0619 13:05:09.753852 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:09.753856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:09.754574 (+ 718us) server_negotiation.cc:407] Connection header received
0619 13:05:09.757145 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:09.757151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:09.757236 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:09.757368 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:09.759576 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:09.760303 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:09.762799 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:09.763235 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:09.763406 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:09.768243 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:09.768248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:09.768252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:09.768287 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:09.771142 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:09.772177 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:09.772181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:09.772183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:09.772312 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:09.772894 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:09.772898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:09.772900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:09.773181 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:09.773291 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:09.774214 (+ 923us) server_negotiation.cc:299] Negotiation successful
0619 13:05:09.774409 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":245,"threads_started":1}
I20260619 13:05:09.776285 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:09.753758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45025 (local address 127.1.87.66:36185)
0619 13:05:09.754415 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:09.754455 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:09.756865 (+ 2410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:09.757623 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:09.757636 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:09.758488 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:09.759365 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:09.759395 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:09.760482 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:09.760491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:09.762627 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:09.762651 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:09.763508 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:09.763518 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:09.763873 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:09.764732 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:09.764768 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:09.767951 (+ 3183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:09.771305 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:09.771345 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:09.771367 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:09.772032 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:09.772442 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:09.772450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:09.772456 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:09.772752 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:09.773308 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:09.773318 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:09.773790 (+ 472us) client_negotiation.cc:769] Sending connection context
0619 13:05:09.774108 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 13:05:09.774742 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":266,"threads_started":1}
I20260619 13:05:09.777875 19938 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45025
I20260619 13:05:09.778327 19938 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:09.779039 19938 heartbeater.cc:511] Master 127.1.87.126:45025 requested a full tablet report, sending...
I20260619 13:05:09.780721 19615 ts_manager.cc:194] Registered new tserver with Master: d19af0f7119c46a5bbc977ee3e9f6ac2 (127.1.87.66:35771)
I20260619 13:05:09.782040 19615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:36185
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:05:09.814023 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45025
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:10.142637 19944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:10.142913 19944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:10.143103 19944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:10.150700 19944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:10.150817 19944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:10.151015 19944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:05:10.160989 19944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:45025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:10.162264 19944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:10.163628 19944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:10.176559 19952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:10.176623 19950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:10.176604 19949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:10.182866 19944 server_base.cc:1034] running on GCE node
I20260619 13:05:10.183568 19944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:10.184537 19944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:10.185762 19944 hybrid_clock.cc:648] HybridClock initialized: now 1781874310185714 us; error 33 us; skew 500 ppm
I20260619 13:05:10.186116 19944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:10 dist-test-slave-tpsm krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:10.189391 19944 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:05:10.190923 19944 webserver.cc:460] Webserver started at http://127.1.87.67:34849/ using document root <none> and password file <none>
I20260619 13:05:10.191686 19944 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:10.191830 19944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:10.192142 19944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:10.195119 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "c63d7359431c4266a1d6cbcd0ea4afcf"
format_stamp: "Formatted at 2026-06-19 13:05:10 on dist-test-slave-tpsm"
server_key: "d2d97fb2b511f59c971c33b068aef2a4"
server_key_iv: "7f5fe0af2b3151fdd8075eeef404e5a1"
server_key_version: "encryptionkey@0"
I20260619 13:05:10.196048 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "c63d7359431c4266a1d6cbcd0ea4afcf"
format_stamp: "Formatted at 2026-06-19 13:05:10 on dist-test-slave-tpsm"
server_key: "d2d97fb2b511f59c971c33b068aef2a4"
server_key_iv: "7f5fe0af2b3151fdd8075eeef404e5a1"
server_key_version: "encryptionkey@0"
I20260619 13:05:10.201269 19944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:05:10.204599 19959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:10.205884 19944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:10.206050 19944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "c63d7359431c4266a1d6cbcd0ea4afcf"
format_stamp: "Formatted at 2026-06-19 13:05:10 on dist-test-slave-tpsm"
server_key: "d2d97fb2b511f59c971c33b068aef2a4"
server_key_iv: "7f5fe0af2b3151fdd8075eeef404e5a1"
server_key_version: "encryptionkey@0"
I20260619 13:05:10.206254 19944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:10.218948 19944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:10.223433 19944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:10.223732 19944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:10.224735 19944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:10.226063 19944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:10.226171 19944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:10.226318 19944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:10.226390 19944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:10.260962 19804 heartbeater.cc:503] Master 127.1.87.126:45025 was elected leader, sending a full tablet report...
I20260619 13:05:10.261511 19944 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:41589
I20260619 13:05:10.261531 20071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:41589 every 8 connection(s)
I20260619 13:05:10.263088 19944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:05:10.268586 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 19944
I20260619 13:05:10.268761 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:05:10.269341 1373 external_mini_cluster.cc:1428] Setting key f8f355989f3bdfb6bd36199a4284d88e
Jun 19 13:05:10 dist-test-slave-tpsm krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:10.282812 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.265747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:36863 (local address 127.1.87.126:45025)
0619 13:05:10.265956 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.265960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.267065 (+ 1105us) server_negotiation.cc:407] Connection header received
0619 13:05:10.268941 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.268947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.269061 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.269179 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:10.270861 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.271805 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.273351 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.273738 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.273842 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.277267 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:10.277272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:10.277275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:10.277309 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:10.280042 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.280751 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.280754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.280756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.280874 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.281240 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.281243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.281245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.281502 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:10.281612 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.282266 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.282491 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:05:10.283903 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.266161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:45025 (local address 127.1.87.67:36863)
0619 13:05:10.266878 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:10.266919 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:10.268605 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:10.269467 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:10.269482 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:10.269998 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:10.270687 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.270704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.271966 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.271973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.273210 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.273222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.273967 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.273973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.274197 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.274929 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:10.274951 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:10.277080 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:10.280189 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:10.280196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:10.280209 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:10.280634 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:10.280965 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:10.280968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:10.280971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:10.281146 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:10.281619 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:10.281625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:10.281927 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 13:05:10.282164 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 13:05:10.282533 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":330,"threads_started":1}
I20260619 13:05:10.285323 20072 heartbeater.cc:347] Connected to a master server at 127.1.87.126:45025
I20260619 13:05:10.285640 20072 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:10.286345 20072 heartbeater.cc:511] Master 127.1.87.126:45025 requested a full tablet report, sending...
I20260619 13:05:10.287933 19615 ts_manager.cc:194] Registered new tserver with Master: c63d7359431c4266a1d6cbcd0ea4afcf (127.1.87.67:41589)
I20260619 13:05:10.288713 19615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:36863
I20260619 13:05:10.297232 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:05:10.316004 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.301450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.1.87.126:45025)
0619 13:05:10.301658 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.301668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.302001 (+ 333us) server_negotiation.cc:407] Connection header received
0619 13:05:10.302240 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.302244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.302313 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.302425 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:10.303968 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.304728 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.306193 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.306716 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.306861 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.309936 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:10.309942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:10.309946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:10.309981 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:10.312773 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.313623 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.313628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.313631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.313778 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.314292 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.314303 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.314305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.314630 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:10.314960 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.315467 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.315669 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:05:10.319766 19615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:05:10.326555 19615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:05:10.355857 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.338611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:41589 (local address 127.0.0.1:47102)
0619 13:05:10.339566 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:10.339615 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:10.339791 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:10.345904 (+ 6113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:10.345915 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:10.345967 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:10.346546 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.346563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.350382 (+ 3819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.350389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.352831 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.352848 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.354754 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.354760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.354991 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.355055 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 13:05:10.355199 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 13:05:10.355413 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":6,"thread_start_us":6425,"threads_started":1}
I20260619 13:05:10.357048 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.345429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:35771 (local address 127.0.0.1:38016)
0619 13:05:10.349381 (+ 3952us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:10.349399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:10.349505 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:10.350514 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:10.350520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:10.350551 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:10.351021 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.351032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.352973 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.352980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.354415 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.354426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.356268 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.356275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.356422 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.356445 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:05:10.356531 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 13:05:10.356689 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3815,"thread_start_us":3837,"threads_started":1}
I20260619 13:05:10.357439 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.338934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.1.87.67:41589)
0619 13:05:10.345081 (+ 6147us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.345094 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.345135 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:05:10.345262 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.345273 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.345594 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.346811 (+ 1217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:10.348580 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.350215 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.353045 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.354534 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.354671 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.354898 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.356906 (+ 2008us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.357074 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5918,"thread_start_us":5714,"threads_started":1}
I20260619 13:05:10.357447 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.338874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39163 (local address 127.0.0.1:49426)
0619 13:05:10.345296 (+ 6422us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:10.345317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:10.349615 (+ 4298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:10.350252 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:10.350256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:10.350281 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:10.350768 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.350778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.352766 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.352771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.355085 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.355096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.356620 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.356626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.356808 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.356839 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:05:10.356929 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 13:05:10.357077 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6289,"mutex_wait_us":103,"thread_start_us":859,"threads_started":1}
I20260619 13:05:10.358119 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.339538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.1.87.65:39163)
0619 13:05:10.349211 (+ 9673us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.349223 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.349751 (+ 528us) server_negotiation.cc:407] Connection header received
0619 13:05:10.349852 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.349861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.350074 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.350264 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:10.350960 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.352619 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.355248 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.356416 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.356552 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.356795 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.357633 (+ 838us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.357801 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9439,"thread_start_us":9256,"threads_started":1}
I20260619 13:05:10.358338 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.345755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.1.87.66:35771)
0619 13:05:10.349970 (+ 4215us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.349978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.350001 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:05:10.350081 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.350087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.350313 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.351132 (+ 819us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:10.351372 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.352836 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.354570 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.356041 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.357600 (+ 1559us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.357789 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.357849 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.358011 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4066,"thread_start_us":3936,"threads_started":1}
I20260619 13:05:10.361135 20007 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 (DEFAULT_TABLE table=test-table [id=d490c74526584150857a7efa9a649010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:10.361449 19873 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 (DEFAULT_TABLE table=test-table [id=d490c74526584150857a7efa9a649010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:10.361443 19739 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 (DEFAULT_TABLE table=test-table [id=d490c74526584150857a7efa9a649010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:10.362411 20007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5f8620b15e48a992cbcbe5bb3087c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:10.362679 19873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5f8620b15e48a992cbcbe5bb3087c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:10.362692 19739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5f8620b15e48a992cbcbe5bb3087c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:10.373876 20090 tablet_bootstrap.cc:492] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: Bootstrap starting.
I20260619 13:05:10.376873 20090 tablet_bootstrap.cc:654] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:10.377564 20091 tablet_bootstrap.cc:492] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: Bootstrap starting.
I20260619 13:05:10.378288 20090 log.cc:824] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:10.379149 20092 tablet_bootstrap.cc:492] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: Bootstrap starting.
I20260619 13:05:10.381233 20091 tablet_bootstrap.cc:654] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:10.381943 20092 tablet_bootstrap.cc:654] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:10.382716 20091 log.cc:824] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:10.382867 20092 log.cc:824] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:10.385492 20092 tablet_bootstrap.cc:492] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: No bootstrap required, opened a new log
I20260619 13:05:10.385598 20091 tablet_bootstrap.cc:492] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: No bootstrap required, opened a new log
I20260619 13:05:10.385600 20090 tablet_bootstrap.cc:492] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: No bootstrap required, opened a new log
I20260619 13:05:10.385814 20092 ts_tablet_manager.cc:1397] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260619 13:05:10.385855 20090 ts_tablet_manager.cc:1397] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260619 13:05:10.385856 20091 ts_tablet_manager.cc:1397] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 13:05:10.393285 20092 raft_consensus.cc:348] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } }
I20260619 13:05:10.393678 20092 raft_consensus.cc:374] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:10.393796 20092 raft_consensus.cc:729] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074047f5005c4d1d8c04018f60571860, State: Initialized, Role: FOLLOWER
I20260619 13:05:10.394418 20092 consensus_queue.cc:260] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } }
I20260619 13:05:10.396311 20092 ts_tablet_manager.cc:1428] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20260619 13:05:10.396595 20090 raft_consensus.cc:348] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } }
I20260619 13:05:10.396627 20091 raft_consensus.cc:348] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } }
I20260619 13:05:10.397053 20091 raft_consensus.cc:374] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:10.397053 20090 raft_consensus.cc:374] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:10.397156 20090 raft_consensus.cc:729] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63d7359431c4266a1d6cbcd0ea4afcf, State: Initialized, Role: FOLLOWER
I20260619 13:05:10.397202 20091 raft_consensus.cc:729] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19af0f7119c46a5bbc977ee3e9f6ac2, State: Initialized, Role: FOLLOWER
I20260619 13:05:10.397784 20090 consensus_queue.cc:260] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } }
I20260619 13:05:10.397791 20091 consensus_queue.cc:260] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } }
I20260619 13:05:10.399531 19938 heartbeater.cc:503] Master 127.1.87.126:45025 was elected leader, sending a full tablet report...
I20260619 13:05:10.399973 20090 ts_tablet_manager.cc:1428] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260619 13:05:10.400802 20072 heartbeater.cc:503] Master 127.1.87.126:45025 was elected leader, sending a full tablet report...
I20260619 13:05:10.402669 20091 ts_tablet_manager.cc:1428] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
W20260619 13:05:10.477111 19805 tablet.cc:2369] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:05:10.492638 20098 raft_consensus.cc:482] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:05:10.492930 20098 raft_consensus.cc:504] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } }
I20260619 13:05:10.495149 20098 leader_election.cc:288] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63d7359431c4266a1d6cbcd0ea4afcf (127.1.87.67:41589), 074047f5005c4d1d8c04018f60571860 (127.1.87.65:39163)
I20260619 13:05:10.502592 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.495599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:56389 (local address 127.1.87.65:39163)
0619 13:05:10.495849 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.495856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.496321 (+ 465us) server_negotiation.cc:407] Connection header received
0619 13:05:10.496502 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.496509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.496574 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.496673 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:10.497297 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.498392 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.500574 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.501578 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.501684 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.501762 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.502223 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.502322 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:05:10.502724 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.495454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:39163 (local address 127.1.87.66:56389)
0619 13:05:10.496204 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:10.496222 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:10.496345 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:10.496699 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:10.496704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:10.496737 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:10.497141 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.497155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.498524 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.498530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.500424 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.500436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.501834 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.501842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.502036 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.502074 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:05:10.502195 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 13:05:10.502331 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":398,"threads_started":1}
W20260619 13:05:10.502911 19939 tablet.cc:2369] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:05:10.503468 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.496105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:41589 (local address 127.1.87.66:45917)
0619 13:05:10.496898 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:10.496917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:10.497050 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:10.497410 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:10.497417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:10.497445 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:10.497873 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.497885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.499266 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.499275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.501011 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:10.501025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.502650 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.502659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:10.502867 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.502903 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:05:10.502984 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 13:05:10.503112 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":443,"threads_started":1}
I20260619 13:05:10.504185 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.495716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:45917 (local address 127.1.87.67:41589)
0619 13:05:10.496041 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.496047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.497019 (+ 972us) server_negotiation.cc:407] Connection header received
0619 13:05:10.497192 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.497200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.497283 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.497360 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:10.498037 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.499068 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.501161 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.502401 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.502532 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.503888 (+ 1356us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.503932 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.504021 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260619 13:05:10.504318 19759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5f8620b15e48a992cbcbe5bb3087c0" candidate_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074047f5005c4d1d8c04018f60571860" is_pre_election: true
I20260619 13:05:10.504848 19759 raft_consensus.cc:2454] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19af0f7119c46a5bbc977ee3e9f6ac2 in term 0.
I20260619 13:05:10.504990 20027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5f8620b15e48a992cbcbe5bb3087c0" candidate_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" is_pre_election: true
I20260619 13:05:10.505472 19829 leader_election.cc:302] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074047f5005c4d1d8c04018f60571860, d19af0f7119c46a5bbc977ee3e9f6ac2; no voters:
I20260619 13:05:10.505473 20027 raft_consensus.cc:2454] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19af0f7119c46a5bbc977ee3e9f6ac2 in term 0.
I20260619 13:05:10.505849 20098 raft_consensus.cc:2781] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:05:10.506021 20098 raft_consensus.cc:482] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:05:10.506153 20098 raft_consensus.cc:3037] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:10.508667 20098 raft_consensus.cc:504] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } }
I20260619 13:05:10.509333 20098 leader_election.cc:288] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [CANDIDATE]: Term 1 election: Requested vote from peers c63d7359431c4266a1d6cbcd0ea4afcf (127.1.87.67:41589), 074047f5005c4d1d8c04018f60571860 (127.1.87.65:39163)
I20260619 13:05:10.509788 20027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5f8620b15e48a992cbcbe5bb3087c0" candidate_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf"
I20260619 13:05:10.509857 19759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5f8620b15e48a992cbcbe5bb3087c0" candidate_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074047f5005c4d1d8c04018f60571860"
I20260619 13:05:10.510033 20027 raft_consensus.cc:3037] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:10.510095 19759 raft_consensus.cc:3037] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:10.512369 20027 raft_consensus.cc:2454] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19af0f7119c46a5bbc977ee3e9f6ac2 in term 1.
I20260619 13:05:10.512490 19759 raft_consensus.cc:2454] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19af0f7119c46a5bbc977ee3e9f6ac2 in term 1.
I20260619 13:05:10.512861 19828 leader_election.cc:302] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c63d7359431c4266a1d6cbcd0ea4afcf, d19af0f7119c46a5bbc977ee3e9f6ac2; no voters:
I20260619 13:05:10.513146 20098 raft_consensus.cc:2781] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:10.513628 20098 raft_consensus.cc:686] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 1 LEADER]: Becoming Leader. State: Replica: d19af0f7119c46a5bbc977ee3e9f6ac2, State: Running, Role: LEADER
I20260619 13:05:10.514102 20098 consensus_queue.cc:237] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } }
W20260619 13:05:10.514731 20073 tablet.cc:2369] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:05:10.519093 19614 catalog_manager.cc:5495] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d19af0f7119c46a5bbc977ee3e9f6ac2 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19af0f7119c46a5bbc977ee3e9f6ac2" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 35771 } health_report { overall_health: HEALTHY } } }
I20260619 13:05:10.545321 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.540058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38018 (local address 127.1.87.66:35771)
0619 13:05:10.540249 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.540254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.540282 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:05:10.540407 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.540411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.540484 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.540603 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:05:10.540945 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.541634 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.542982 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.543570 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.543680 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.544170 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:05:10.544660 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:05:10.544799 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.544944 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.545015 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:05:10.553625 20027 raft_consensus.cc:1261] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [term 1 FOLLOWER]: Refusing update from remote peer d19af0f7119c46a5bbc977ee3e9f6ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:10.553714 19759 raft_consensus.cc:1261] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [term 1 FOLLOWER]: Refusing update from remote peer d19af0f7119c46a5bbc977ee3e9f6ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:10.554652 20101 consensus_queue.cc:1035] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074047f5005c4d1d8c04018f60571860" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:10.555030 20098 consensus_queue.cc:1035] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63d7359431c4266a1d6cbcd0ea4afcf" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:10.597239 19614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:10.597626 19614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:10.602906 19614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b9ee8b7672b4ce49e35ba1e26e86d42: Sending DeleteTablet for 3 replicas of tablet 1f5f8620b15e48a992cbcbe5bb3087c0
I20260619 13:05:10.604444 19739 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:10 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:49426
I20260619 13:05:10.604611 19738 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:10 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:49426
I20260619 13:05:10.604887 20007 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:10 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:47102
I20260619 13:05:10.605130 19873 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:10 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:38016
I20260619 13:05:10.605758 19872 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:10 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:38016
I20260619 13:05:10.605947 20114 tablet_replica.cc:330] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: stopping tablet replica
I20260619 13:05:10.608070 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 19676
I20260619 13:05:10.616282 20116 tablet_replica.cc:330] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: stopping tablet replica
I20260619 13:05:10.616891 20116 raft_consensus.cc:2229] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:05:10.617667 20116 raft_consensus.cc:2258] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:10.618744 20115 tablet_replica.cc:330] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: stopping tablet replica
I20260619 13:05:10.606509 20114 raft_consensus.cc:2229] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:10.619437 20114 raft_consensus.cc:2258] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:10.620366 20116 ts_tablet_manager.cc:1905] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:10.621389 20114 ts_tablet_manager.cc:1905] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:10.621896 20115 raft_consensus.cc:2229] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:10.622443 20115 raft_consensus.cc:2258] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:10.628360 20115 ts_tablet_manager.cc:1905] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:10.628489 20114 ts_tablet_manager.cc:1918] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:10.628665 20114 log.cc:1196] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/1f5f8620b15e48a992cbcbe5bb3087c0
I20260619 13:05:10.629223 20114 ts_tablet_manager.cc:1939] T 1f5f8620b15e48a992cbcbe5bb3087c0 P 074047f5005c4d1d8c04018f60571860: Deleting consensus metadata
Jun 19 13:05:10 dist-test-slave-tpsm krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:05:10.631155 19602 catalog_manager.cc:4841] TS 074047f5005c4d1d8c04018f60571860 (127.1.87.65:39163): tablet 1f5f8620b15e48a992cbcbe5bb3087c0 (table test-table [id=d490c74526584150857a7efa9a649010]) successfully deleted
W20260619 13:05:10.632295 19602 catalog_manager.cc:4808] TS 074047f5005c4d1d8c04018f60571860 (127.1.87.65:39163): delete failed for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 because the tablet was not found. No further retry: Not found: Tablet not found: 1f5f8620b15e48a992cbcbe5bb3087c0
I20260619 13:05:10.635409 20116 ts_tablet_manager.cc:1918] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:10.635659 20116 log.cc:1196] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/1f5f8620b15e48a992cbcbe5bb3087c0
I20260619 13:05:10.636291 20116 ts_tablet_manager.cc:1939] T 1f5f8620b15e48a992cbcbe5bb3087c0 P d19af0f7119c46a5bbc977ee3e9f6ac2: Deleting consensus metadata
I20260619 13:05:10.637501 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.612545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.1.87.65:39163)
0619 13:05:10.612844 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.612850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.619072 (+ 6222us) server_negotiation.cc:407] Connection header received
0619 13:05:10.619255 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.619260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.619418 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.619536 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:10.621310 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.622417 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.624304 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.624973 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.625071 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.630618 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:10.630631 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:10.630648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:10.630697 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:10.633840 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.634539 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.634547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.634551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.634703 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.635309 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.635340 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.635346 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.635786 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:10.636198 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.636846 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.637107 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260619 13:05:10.638335 19601 catalog_manager.cc:4841] TS d19af0f7119c46a5bbc977ee3e9f6ac2 (127.1.87.66:35771): tablet 1f5f8620b15e48a992cbcbe5bb3087c0 (table test-table [id=d490c74526584150857a7efa9a649010]) successfully deleted
I20260619 13:05:10.639070 19799 generic_service.cc:196] Checking for leaks (request via RPC)
W20260619 13:05:10.639302 19601 catalog_manager.cc:4808] TS d19af0f7119c46a5bbc977ee3e9f6ac2 (127.1.87.66:35771): delete failed for tablet 1f5f8620b15e48a992cbcbe5bb3087c0 because the tablet was not found. No further retry: Not found: Tablet not found: 1f5f8620b15e48a992cbcbe5bb3087c0
I20260619 13:05:10.640177 20115 ts_tablet_manager.cc:1918] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:10.640398 20115 log.cc:1196] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/1f5f8620b15e48a992cbcbe5bb3087c0
I20260619 13:05:10.640981 20115 ts_tablet_manager.cc:1939] T 1f5f8620b15e48a992cbcbe5bb3087c0 P c63d7359431c4266a1d6cbcd0ea4afcf: Deleting consensus metadata
I20260619 13:05:10.643009 19601 catalog_manager.cc:4841] TS c63d7359431c4266a1d6cbcd0ea4afcf (127.1.87.67:41589): tablet 1f5f8620b15e48a992cbcbe5bb3087c0 (table test-table [id=d490c74526584150857a7efa9a649010]) successfully deleted
I20260619 13:05:10.772270 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 19676
I20260619 13:05:10.800297 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 19810
Jun 19 13:05:10 dist-test-slave-tpsm krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:05:10.824200 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.801133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.1.87.66:35771)
0619 13:05:10.801726 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.801733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.801755 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:05:10.801818 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.801824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.801902 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.802002 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:10.804240 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.805573 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.808146 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.809030 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.809194 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.815376 (+ 6182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:10.815389 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:10.815409 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:10.815471 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:10.819903 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.821396 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.821411 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.821415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.821633 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.822216 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.822221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.822225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.822719 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:10.823048 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.823455 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.823717 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450}
I20260619 13:05:10.825107 19933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:10.938335 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 19810
I20260619 13:05:10.957101 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 19944
Jun 19 13:05:10 dist-test-slave-tpsm krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:05:10.972435 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:10.957864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.1.87.67:41589)
0619 13:05:10.958059 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:10.958067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:10.958087 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:05:10.958261 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:10.958273 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:10.958362 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:10.958466 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:10.959911 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.960610 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.961916 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:10.962299 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:10.962432 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:10.966942 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:10.966957 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:10.966974 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:10.967024 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:10.969778 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.970452 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.970459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.970461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.970614 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:10.970925 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:10.970929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:10.970930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:10.971413 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:10.971626 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:10.971909 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 13:05:10.972066 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:05:10.973101 20067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:11.064095 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 19944
I20260619 13:05:11.080626 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 19583
I20260619 13:05:11.081765 19645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:11.193406 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 19583
2026-06-19T13:05:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4434 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T13:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:11Z Disabled control of system clock
I20260619 13:05:11.275442 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39593
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:39593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:05:11.608537 20125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:11.608819 20125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:11.608899 20125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:11.616277 20125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:11.616386 20125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:11.616425 20125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:11.616474 20125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:11.616518 20125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:11.625835 20125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:39593
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:11.627281 20125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:11.629251 20125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:11.641009 20130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:11.640933 20131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:11.640978 20133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:11.641849 20125 server_base.cc:1034] running on GCE node
I20260619 13:05:11.642683 20125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:11.644198 20125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:11.645412 20125 hybrid_clock.cc:648] HybridClock initialized: now 1781874311645365 us; error 39 us; skew 500 ppm
I20260619 13:05:11.645782 20125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:11.648104 20125 webserver.cc:460] Webserver started at http://127.1.87.126:43903/ using document root <none> and password file <none>
I20260619 13:05:11.648782 20125 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:11.648895 20125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:11.649225 20125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:11.652729 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "e691a271ad6545419211a6728daf842e"
format_stamp: "Formatted at 2026-06-19 13:05:11 on dist-test-slave-tpsm"
server_key: "5df524ef83909dc1c1a810f3e6991033"
server_key_iv: "a5a1abe164bd2540302d09e6974135ce"
server_key_version: "encryptionkey@0"
I20260619 13:05:11.653601 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "e691a271ad6545419211a6728daf842e"
format_stamp: "Formatted at 2026-06-19 13:05:11 on dist-test-slave-tpsm"
server_key: "5df524ef83909dc1c1a810f3e6991033"
server_key_iv: "a5a1abe164bd2540302d09e6974135ce"
server_key_version: "encryptionkey@0"
I20260619 13:05:11.658821 20125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:05:11.662329 20139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:11.663774 20125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:11.663966 20125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "e691a271ad6545419211a6728daf842e"
format_stamp: "Formatted at 2026-06-19 13:05:11 on dist-test-slave-tpsm"
server_key: "5df524ef83909dc1c1a810f3e6991033"
server_key_iv: "a5a1abe164bd2540302d09e6974135ce"
server_key_version: "encryptionkey@0"
I20260619 13:05:11.664189 20125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:11.684022 20125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:11.684979 20125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:11.685289 20125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:11.706162 20125 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:39593
I20260619 13:05:11.706171 20190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:39593 every 8 connection(s)
I20260619 13:05:11.707562 20125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:11.709076 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 20125
I20260619 13:05:11.709251 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:11.709729 1373 external_mini_cluster.cc:1428] Setting key 77df0ec5a9bab7ebeb823ad9ccb33a19
I20260619 13:05:11.711975 20191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:11.725345 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:11.711657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.1.87.126:39593)
0619 13:05:11.712695 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:11.712709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:11.712757 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:05:11.714934 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:11.714960 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:11.715565 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:11.716011 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:11.716522 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:11.718676 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:11.720132 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:11.721130 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:11.721316 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:11.722474 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:11.722485 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:11.722502 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:11.723233 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:11.723363 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:11.723389 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:11.723507 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:11.723778 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 13:05:11.724110 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":374,"threads_started":1}
I20260619 13:05:11.726531 20191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e: Bootstrap starting.
I20260619 13:05:11.729980 20191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:11.731086 20191 log.cc:824] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:11.733898 20191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e: No bootstrap required, opened a new log
I20260619 13:05:11.741252 20191 raft_consensus.cc:348] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e691a271ad6545419211a6728daf842e" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39593 } }
I20260619 13:05:11.741595 20191 raft_consensus.cc:374] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:11.741710 20191 raft_consensus.cc:729] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e691a271ad6545419211a6728daf842e, State: Initialized, Role: FOLLOWER
I20260619 13:05:11.742237 20191 consensus_queue.cc:260] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e691a271ad6545419211a6728daf842e" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39593 } }
I20260619 13:05:11.742457 20191 raft_consensus.cc:388] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:11.742611 20191 raft_consensus.cc:482] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:11.742789 20191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:11.745071 20191 raft_consensus.cc:504] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e691a271ad6545419211a6728daf842e" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39593 } }
I20260619 13:05:11.745716 20191 leader_election.cc:302] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e691a271ad6545419211a6728daf842e; no voters:
I20260619 13:05:11.746332 20191 leader_election.cc:288] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:11.746474 20196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:11.747265 20196 raft_consensus.cc:686] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [term 1 LEADER]: Becoming Leader. State: Replica: e691a271ad6545419211a6728daf842e, State: Running, Role: LEADER
I20260619 13:05:11.747750 20196 consensus_queue.cc:237] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e691a271ad6545419211a6728daf842e" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39593 } }
I20260619 13:05:11.748485 20191 sys_catalog.cc:564] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:11.750473 20198 sys_catalog.cc:455] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e691a271ad6545419211a6728daf842e. Latest consensus state: current_term: 1 leader_uuid: "e691a271ad6545419211a6728daf842e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e691a271ad6545419211a6728daf842e" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39593 } } }
I20260619 13:05:11.750567 20197 sys_catalog.cc:455] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e691a271ad6545419211a6728daf842e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e691a271ad6545419211a6728daf842e" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39593 } } }
I20260619 13:05:11.750981 20197 sys_catalog.cc:458] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:11.750851 20198 sys_catalog.cc:458] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:11.752835 20202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:11.757931 20202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:11.767220 20202 catalog_manager.cc:1347] Generated new cluster ID: b6d379659adb4b66a1f42378f11fd0b6
I20260619 13:05:11.767367 20202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:11.784070 20202 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:11.785388 20202 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:11.795744 20202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e: Generated new TSK 0
I20260619 13:05:11.796603 20202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:11.803638 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39593
--builtin_ntp_servers=127.1.87.84:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:05:12.170989 20215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:12.171277 20215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:12.171446 20215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:12.178754 20215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:12.178864 20215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:12.178975 20215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:05:12.188509 20215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:12.190294 20215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:12.192233 20215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:12.205386 20220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:12.205400 20221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:12.205488 20223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:12.206208 20215 server_base.cc:1034] running on GCE node
I20260619 13:05:12.206830 20215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:12.207931 20215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:12.209177 20215 hybrid_clock.cc:648] HybridClock initialized: now 1781874312209143 us; error 60 us; skew 500 ppm
I20260619 13:05:12.209703 20215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:12.212273 20215 webserver.cc:460] Webserver started at http://127.1.87.65:39701/ using document root <none> and password file <none>
I20260619 13:05:12.213124 20215 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:12.213390 20215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:12.213753 20215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:12.216810 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "17cb4ad158de4ad6b15e5765894451ef"
format_stamp: "Formatted at 2026-06-19 13:05:12 on dist-test-slave-tpsm"
server_key: "81fbe3a1f2c55f6c4c621a6b4fb15b08"
server_key_iv: "e4c36fb168bc2018770acc4471e2a4c5"
server_key_version: "encryptionkey@0"
I20260619 13:05:12.217664 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "17cb4ad158de4ad6b15e5765894451ef"
format_stamp: "Formatted at 2026-06-19 13:05:12 on dist-test-slave-tpsm"
server_key: "81fbe3a1f2c55f6c4c621a6b4fb15b08"
server_key_iv: "e4c36fb168bc2018770acc4471e2a4c5"
server_key_version: "encryptionkey@0"
I20260619 13:05:12.223459 20215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 13:05:12.227962 20229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:12.229435 20215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:12.229732 20215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "17cb4ad158de4ad6b15e5765894451ef"
format_stamp: "Formatted at 2026-06-19 13:05:12 on dist-test-slave-tpsm"
server_key: "81fbe3a1f2c55f6c4c621a6b4fb15b08"
server_key_iv: "e4c36fb168bc2018770acc4471e2a4c5"
server_key_version: "encryptionkey@0"
I20260619 13:05:12.229979 20215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:12.260398 20215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:12.261308 20215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:12.261598 20215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:12.262586 20215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:12.264287 20215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:12.264406 20215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:12.264559 20215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:12.264650 20215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:12.298215 20215 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:35297
I20260619 13:05:12.298285 20341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:35297 every 8 connection(s)
I20260619 13:05:12.299539 20215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:05:12.305287 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 20215
I20260619 13:05:12.305469 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:05:12.305996 1373 external_mini_cluster.cc:1428] Setting key abd1c98bd8ef754666483041659b7122
I20260619 13:05:12.309126 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39593
--builtin_ntp_servers=127.1.87.84:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:05:12.313165 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:12.301973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:58941 (local address 127.1.87.126:39593)
0619 13:05:12.302425 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:12.302432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:12.303574 (+ 1142us) server_negotiation.cc:407] Connection header received
0619 13:05:12.305541 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:12.305548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:12.305675 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:12.305782 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:12.307013 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:12.307813 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:12.310174 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:12.310627 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:12.310738 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:12.312342 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:12.312347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:12.312350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:12.312368 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:12.312407 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:12.312419 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:12.312499 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:12.312827 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 13:05:12.312903 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
I20260619 13:05:12.314584 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:12.302514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39593 (local address 127.1.87.65:58941)
0619 13:05:12.303423 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:12.303458 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:12.305213 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:12.306046 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:12.306059 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:12.306168 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:12.306840 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:12.306861 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:12.307961 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:12.307973 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:12.310001 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:12.310019 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:12.310861 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:12.310871 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:12.311205 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:12.311987 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:05:12.311993 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:12.312047 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:05:12.312144 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:12.312571 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:12.312585 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:12.312596 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 13:05:12.312812 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 13:05:12.313228 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":271,"threads_started":1}
I20260619 13:05:12.316226 20342 heartbeater.cc:347] Connected to a master server at 127.1.87.126:39593
I20260619 13:05:12.316527 20342 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:12.317265 20342 heartbeater.cc:511] Master 127.1.87.126:39593 requested a full tablet report, sending...
I20260619 13:05:12.319898 20156 ts_manager.cc:194] Registered new tserver with Master: 17cb4ad158de4ad6b15e5765894451ef (127.1.87.65:35297)
I20260619 13:05:12.322031 20156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.65:58941
W20260619 13:05:12.646165 20346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:12.646445 20346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:12.646541 20346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:12.653868 20346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:12.653975 20346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:12.654096 20346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:05:12.663304 20346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:12.664573 20346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:12.666486 20346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:12.679669 20352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:12.679716 20351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:12.679638 20354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:12.680365 20346 server_base.cc:1034] running on GCE node
I20260619 13:05:12.680998 20346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:12.681952 20346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:12.683178 20346 hybrid_clock.cc:648] HybridClock initialized: now 1781874312683138 us; error 39 us; skew 500 ppm
I20260619 13:05:12.683553 20346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:12.685966 20346 webserver.cc:460] Webserver started at http://127.1.87.66:45295/ using document root <none> and password file <none>
I20260619 13:05:12.686733 20346 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:12.686842 20346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:12.687189 20346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:12.690119 20346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "25b5b6ee85e44ac38ed6866d3689849e"
format_stamp: "Formatted at 2026-06-19 13:05:12 on dist-test-slave-tpsm"
server_key: "6b89c36f3763c9ed30c926b0daed2848"
server_key_iv: "0cd92ee91184a7350829db0bfe360e7a"
server_key_version: "encryptionkey@0"
I20260619 13:05:12.690886 20346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "25b5b6ee85e44ac38ed6866d3689849e"
format_stamp: "Formatted at 2026-06-19 13:05:12 on dist-test-slave-tpsm"
server_key: "6b89c36f3763c9ed30c926b0daed2848"
server_key_iv: "0cd92ee91184a7350829db0bfe360e7a"
server_key_version: "encryptionkey@0"
I20260619 13:05:12.695784 20346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:05:12.699165 20360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:12.700461 20346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:12.700634 20346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "25b5b6ee85e44ac38ed6866d3689849e"
format_stamp: "Formatted at 2026-06-19 13:05:12 on dist-test-slave-tpsm"
server_key: "6b89c36f3763c9ed30c926b0daed2848"
server_key_iv: "0cd92ee91184a7350829db0bfe360e7a"
server_key_version: "encryptionkey@0"
I20260619 13:05:12.700851 20346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:12.725797 20346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:12.726805 20346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:12.727187 20346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:12.728385 20346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:12.730026 20346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:12.730127 20346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:12.730218 20346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:12.730281 20346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:12.766392 20346 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:40999
I20260619 13:05:12.766450 20472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:40999 every 8 connection(s)
I20260619 13:05:12.767916 20346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:05:12.778095 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 20346
I20260619 13:05:12.778270 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:05:12.778766 1373 external_mini_cluster.cc:1428] Setting key 41a3e9451d49e3c71ae30c9af0c70262
I20260619 13:05:12.780426 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:12.770604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:45591 (local address 127.1.87.126:39593)
0619 13:05:12.770789 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:12.770793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:12.772098 (+ 1305us) server_negotiation.cc:407] Connection header received
0619 13:05:12.773864 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:12.773872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:12.773966 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:12.774077 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:12.775116 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:12.775902 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:12.777414 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:12.777809 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:12.777945 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:12.779595 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:12.779600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:12.779604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:12.779623 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:12.779656 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:12.779668 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:12.779775 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:12.779997 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 13:05:12.780096 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:05:12.781780 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:12.771038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39593 (local address 127.1.87.66:45591)
0619 13:05:12.771943 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:12.771986 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:12.773602 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:12.774298 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:12.774306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:12.774388 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:12.774954 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:12.774970 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:12.776045 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:12.776050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:12.777267 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:12.777278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:12.778031 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:12.778038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:12.778368 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:12.779138 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:05:12.779141 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:12.779210 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:05:12.779360 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:12.779774 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:12.779786 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:12.779796 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:05:12.779987 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 13:05:12.780404 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":319,"threads_started":1}
I20260619 13:05:12.781832 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39593
--builtin_ntp_servers=127.1.87.84:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:05:12.783236 20473 heartbeater.cc:347] Connected to a master server at 127.1.87.126:39593
I20260619 13:05:12.783568 20473 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:12.784328 20473 heartbeater.cc:511] Master 127.1.87.126:39593 requested a full tablet report, sending...
I20260619 13:05:12.785784 20156 ts_manager.cc:194] Registered new tserver with Master: 25b5b6ee85e44ac38ed6866d3689849e (127.1.87.66:40999)
I20260619 13:05:12.786605 20156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.66:45591
W20260619 13:05:13.109532 20476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:13.109800 20476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:13.109946 20476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:13.117005 20476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:13.117100 20476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:13.117280 20476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:05:13.126219 20476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:39593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:13.127511 20476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:13.129235 20476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:13.142663 20481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:13.142717 20482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:13.142684 20484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:13.143651 20476 server_base.cc:1034] running on GCE node
I20260619 13:05:13.144299 20476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:13.145363 20476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:13.146639 20476 hybrid_clock.cc:648] HybridClock initialized: now 1781874313146575 us; error 44 us; skew 500 ppm
I20260619 13:05:13.147001 20476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:13.149559 20476 webserver.cc:460] Webserver started at http://127.1.87.67:38117/ using document root <none> and password file <none>
I20260619 13:05:13.150267 20476 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:13.150410 20476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:13.150760 20476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:13.153829 20476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "cbe0d98735c34b1da7c4a90c93485537"
format_stamp: "Formatted at 2026-06-19 13:05:13 on dist-test-slave-tpsm"
server_key: "851e4db11df22c49c6dbea25b066cc9b"
server_key_iv: "2bbdae8bfcedd4be32858bd31a6461c8"
server_key_version: "encryptionkey@0"
I20260619 13:05:13.154695 20476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "cbe0d98735c34b1da7c4a90c93485537"
format_stamp: "Formatted at 2026-06-19 13:05:13 on dist-test-slave-tpsm"
server_key: "851e4db11df22c49c6dbea25b066cc9b"
server_key_iv: "2bbdae8bfcedd4be32858bd31a6461c8"
server_key_version: "encryptionkey@0"
I20260619 13:05:13.159981 20476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 13:05:13.163605 20490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:13.164969 20476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:05:13.165166 20476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "cbe0d98735c34b1da7c4a90c93485537"
format_stamp: "Formatted at 2026-06-19 13:05:13 on dist-test-slave-tpsm"
server_key: "851e4db11df22c49c6dbea25b066cc9b"
server_key_iv: "2bbdae8bfcedd4be32858bd31a6461c8"
server_key_version: "encryptionkey@0"
I20260619 13:05:13.165410 20476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:13.183691 20476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:13.184520 20476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:13.184823 20476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:13.185830 20476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:13.187377 20476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:13.187518 20476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:13.187664 20476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:13.187760 20476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:13.222894 20476 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:35841
I20260619 13:05:13.222944 20602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:35841 every 8 connection(s)
I20260619 13:05:13.224385 20476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:05:13.226562 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 20476
I20260619 13:05:13.226718 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:05:13.227250 1373 external_mini_cluster.cc:1428] Setting key af34679b37d80663ecf1c00f9a4ce6b1
I20260619 13:05:13.237015 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.227147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:46563 (local address 127.1.87.126:39593)
0619 13:05:13.227391 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.227397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.228911 (+ 1514us) server_negotiation.cc:407] Connection header received
0619 13:05:13.230924 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.230928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.230990 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.231072 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:13.232128 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.232851 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.234487 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.234898 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.235082 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.236337 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:13.236342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:13.236345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:13.236362 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:13.236393 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:13.236405 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:13.236477 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.236702 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.236775 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:05:13.237970 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.227616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:39593 (local address 127.1.87.67:46563)
0619 13:05:13.228752 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:13.228805 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:13.230691 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:13.231256 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:13.231263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:13.231406 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:13.231973 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.231991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.232986 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.232989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.234352 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.234364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.235180 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.235187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.235432 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.236059 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:05:13.236062 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:13.236097 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:05:13.236165 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:13.236495 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:13.236503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:13.236508 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:05:13.236653 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 13:05:13.236924 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":365,"threads_started":1}
I20260619 13:05:13.239439 20603 heartbeater.cc:347] Connected to a master server at 127.1.87.126:39593
I20260619 13:05:13.239794 20603 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:13.240584 20603 heartbeater.cc:511] Master 127.1.87.126:39593 requested a full tablet report, sending...
I20260619 13:05:13.242331 20155 ts_manager.cc:194] Registered new tserver with Master: cbe0d98735c34b1da7c4a90c93485537 (127.1.87.67:35841)
I20260619 13:05:13.243189 20155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.67:46563
I20260619 13:05:13.254907 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:05:13.265223 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.259278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.1.87.126:39593)
0619 13:05:13.259594 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.259603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.259953 (+ 350us) server_negotiation.cc:407] Connection header received
0619 13:05:13.260075 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.260079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.260160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.260244 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:13.260741 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.261445 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.262801 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.263212 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.263378 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.264475 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:13.264479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:13.264482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:13.264500 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:13.264532 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:13.264543 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:13.264652 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.264802 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.264881 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 13:05:13.269605 20155 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:05:13.276597 20155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:05:13.318783 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.295502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.1.87.65:35297)
0619 13:05:13.309165 (+ 13663us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.309180 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.309226 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:05:13.309409 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.309424 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.309794 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.310082 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:13.311307 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.313124 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.315480 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.317516 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.317665 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.317947 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.318243 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.318417 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13411,"thread_start_us":13227,"threads_started":1}
I20260619 13:05:13.318854 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.295374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:35297 (local address 127.0.0.1:52824)
0619 13:05:13.306362 (+ 10988us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:13.306415 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:13.306607 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:13.310464 (+ 3857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:13.310473 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:13.310518 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:13.311067 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.311086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.313292 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.313298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.315181 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.315204 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.317711 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.317718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.317953 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.318015 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 13:05:13.318177 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 13:05:13.318385 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10705,"thread_start_us":10646,"threads_started":1}
I20260619 13:05:13.320068 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.295487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:35841 (local address 127.0.0.1:56414)
0619 13:05:13.307969 (+ 12482us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:13.308006 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:13.308189 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:13.311195 (+ 3006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:13.311199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:13.311224 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:13.311599 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.311612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.314664 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.314682 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.317237 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.317251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.319221 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.319229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.319447 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.319485 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:05:13.319570 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 13:05:13.319719 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12280,"mutex_wait_us":239,"thread_start_us":15376,"threads_started":1}
I20260619 13:05:13.320370 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.295968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.1.87.67:35841)
0619 13:05:13.310398 (+ 14430us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.310405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.310439 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:05:13.310533 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.310540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.310731 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.310902 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:13.311804 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.314471 (+ 2667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.317492 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.318877 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.319018 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.319274 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.319667 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.319844 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14246,"thread_start_us":16484,"threads_started":1}
I20260619 13:05:13.320531 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.295946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40999 (local address 127.0.0.1:57894)
0619 13:05:13.310666 (+ 14720us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:13.310691 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:13.311152 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:13.312142 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:13.312148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:13.312175 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:13.312631 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.312646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.314587 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.314593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.317086 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.317106 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.319500 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.319507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.319710 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.319741 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:05:13.319825 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 13:05:13.319953 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14574,"mutex_wait_us":117,"thread_start_us":12370,"threads_started":1}
I20260619 13:05:13.321295 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.296542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.1.87.66:40999)
0619 13:05:13.305653 (+ 9111us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.305668 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.310817 (+ 5149us) server_negotiation.cc:407] Connection header received
0619 13:05:13.311392 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.311403 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.311637 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.311814 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:13.312846 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.314415 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.317377 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.318973 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.319098 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.319361 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.320157 (+ 796us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.320954 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8853,"thread_start_us":8718,"threads_started":1}
I20260619 13:05:13.324009 20408 tablet_service.cc:1467] Processing CreateTablet for tablet a9d020b0a03f433d8dcbc25742898ed6 (DEFAULT_TABLE table=test-table [id=d65d4b5ee39e4271a77f7771353a75f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:13.324123 20277 tablet_service.cc:1467] Processing CreateTablet for tablet a9d020b0a03f433d8dcbc25742898ed6 (DEFAULT_TABLE table=test-table [id=d65d4b5ee39e4271a77f7771353a75f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:13.324575 20538 tablet_service.cc:1467] Processing CreateTablet for tablet a9d020b0a03f433d8dcbc25742898ed6 (DEFAULT_TABLE table=test-table [id=d65d4b5ee39e4271a77f7771353a75f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:13.325268 20408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d020b0a03f433d8dcbc25742898ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:13.325707 20342 heartbeater.cc:503] Master 127.1.87.126:39593 was elected leader, sending a full tablet report...
I20260619 13:05:13.326028 20538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d020b0a03f433d8dcbc25742898ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:13.326041 20277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d020b0a03f433d8dcbc25742898ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:13.337937 20621 tablet_bootstrap.cc:492] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: Bootstrap starting.
I20260619 13:05:13.340863 20622 tablet_bootstrap.cc:492] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: Bootstrap starting.
I20260619 13:05:13.343655 20621 tablet_bootstrap.cc:654] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:13.344092 20623 tablet_bootstrap.cc:492] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: Bootstrap starting.
I20260619 13:05:13.345345 20622 tablet_bootstrap.cc:654] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:13.345705 20621 log.cc:824] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:13.346652 20622 log.cc:824] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:13.347497 20623 tablet_bootstrap.cc:654] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:13.348589 20623 log.cc:824] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:13.349112 20621 tablet_bootstrap.cc:492] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: No bootstrap required, opened a new log
I20260619 13:05:13.349476 20621 ts_tablet_manager.cc:1397] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260619 13:05:13.349700 20622 tablet_bootstrap.cc:492] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: No bootstrap required, opened a new log
I20260619 13:05:13.349982 20622 ts_tablet_manager.cc:1397] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:05:13.351897 20623 tablet_bootstrap.cc:492] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: No bootstrap required, opened a new log
I20260619 13:05:13.352138 20623 ts_tablet_manager.cc:1397] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260619 13:05:13.359284 20623 raft_consensus.cc:348] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } }
I20260619 13:05:13.359755 20623 raft_consensus.cc:374] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:13.359851 20623 raft_consensus.cc:729] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe0d98735c34b1da7c4a90c93485537, State: Initialized, Role: FOLLOWER
I20260619 13:05:13.360481 20623 consensus_queue.cc:260] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } }
I20260619 13:05:13.360600 20621 raft_consensus.cc:348] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } }
I20260619 13:05:13.360612 20622 raft_consensus.cc:348] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } }
I20260619 13:05:13.361029 20622 raft_consensus.cc:374] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:13.361030 20621 raft_consensus.cc:374] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:13.361133 20622 raft_consensus.cc:729] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17cb4ad158de4ad6b15e5765894451ef, State: Initialized, Role: FOLLOWER
I20260619 13:05:13.361133 20621 raft_consensus.cc:729] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b5b6ee85e44ac38ed6866d3689849e, State: Initialized, Role: FOLLOWER
I20260619 13:05:13.361750 20603 heartbeater.cc:503] Master 127.1.87.126:39593 was elected leader, sending a full tablet report...
I20260619 13:05:13.361804 20622 consensus_queue.cc:260] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } }
I20260619 13:05:13.362951 20623 ts_tablet_manager.cc:1428] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260619 13:05:13.363528 20621 consensus_queue.cc:260] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } }
I20260619 13:05:13.365603 20473 heartbeater.cc:503] Master 127.1.87.126:39593 was elected leader, sending a full tablet report...
I20260619 13:05:13.369922 20622 ts_tablet_manager.cc:1428] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260619 13:05:13.370358 20621 ts_tablet_manager.cc:1428] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: Time spent starting tablet: real 0.021s user 0.013s sys 0.003s
I20260619 13:05:13.402099 20627 raft_consensus.cc:482] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:05:13.402412 20627 raft_consensus.cc:504] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } }
I20260619 13:05:13.404215 20627 leader_election.cc:288] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17cb4ad158de4ad6b15e5765894451ef (127.1.87.65:35297), 25b5b6ee85e44ac38ed6866d3689849e (127.1.87.66:40999)
I20260619 13:05:13.412212 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.404910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:55963 (local address 127.1.87.66:40999)
0619 13:05:13.405155 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.405161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.405183 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:05:13.405261 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.405266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.405399 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.405480 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:13.406008 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.406959 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.409507 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.411050 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.411208 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.411385 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.411832 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.411977 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 13:05:13.412207 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.404501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:40999 (local address 127.1.87.67:55963)
0619 13:05:13.404686 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:13.404709 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:13.404842 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:13.405548 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:13.405553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:13.405633 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:13.405818 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.405826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.407140 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.407144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.409310 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.409332 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.411221 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.411227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.411450 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.411495 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 13:05:13.411601 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:05:13.411834 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":26,"spinlock_wait_cycles":23808}
I20260619 13:05:13.412550 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.404769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:35599 (local address 127.1.87.65:35297)
0619 13:05:13.405111 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.405119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.405568 (+ 449us) server_negotiation.cc:407] Connection header received
0619 13:05:13.405731 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.405735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.405800 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.405872 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:13.406399 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.407464 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.409826 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.411482 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.411627 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.411747 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.412117 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.412256 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260619 13:05:13.412971 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.404715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:35297 (local address 127.1.87.67:35599)
0619 13:05:13.405428 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:13.405441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:13.405583 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:13.405921 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:13.405928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:13.405956 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:13.406241 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.406249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.407628 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.407633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.409636 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:13.409645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.411752 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.411761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:13.411918 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.411943 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 13:05:13.412076 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 13:05:13.412215 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":23,"thread_start_us":396,"threads_started":1}
I20260619 13:05:13.413400 20428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d020b0a03f433d8dcbc25742898ed6" candidate_uuid: "cbe0d98735c34b1da7c4a90c93485537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b5b6ee85e44ac38ed6866d3689849e" is_pre_election: true
I20260619 13:05:13.414088 20428 raft_consensus.cc:2454] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbe0d98735c34b1da7c4a90c93485537 in term 0.
I20260619 13:05:13.414053 20297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d020b0a03f433d8dcbc25742898ed6" candidate_uuid: "cbe0d98735c34b1da7c4a90c93485537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cb4ad158de4ad6b15e5765894451ef" is_pre_election: true
I20260619 13:05:13.414640 20297 raft_consensus.cc:2454] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbe0d98735c34b1da7c4a90c93485537 in term 0.
I20260619 13:05:13.414862 20492 leader_election.cc:302] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b5b6ee85e44ac38ed6866d3689849e, cbe0d98735c34b1da7c4a90c93485537; no voters:
I20260619 13:05:13.415385 20627 raft_consensus.cc:2781] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:05:13.415561 20627 raft_consensus.cc:482] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:05:13.415696 20627 raft_consensus.cc:3037] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:13.417878 20627 raft_consensus.cc:504] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } }
I20260619 13:05:13.418577 20627 leader_election.cc:288] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [CANDIDATE]: Term 1 election: Requested vote from peers 17cb4ad158de4ad6b15e5765894451ef (127.1.87.65:35297), 25b5b6ee85e44ac38ed6866d3689849e (127.1.87.66:40999)
I20260619 13:05:13.418969 20297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d020b0a03f433d8dcbc25742898ed6" candidate_uuid: "cbe0d98735c34b1da7c4a90c93485537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cb4ad158de4ad6b15e5765894451ef"
I20260619 13:05:13.419155 20297 raft_consensus.cc:3037] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:13.419212 20428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d020b0a03f433d8dcbc25742898ed6" candidate_uuid: "cbe0d98735c34b1da7c4a90c93485537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b5b6ee85e44ac38ed6866d3689849e"
I20260619 13:05:13.419484 20428 raft_consensus.cc:3037] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:13.421679 20297 raft_consensus.cc:2454] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbe0d98735c34b1da7c4a90c93485537 in term 1.
I20260619 13:05:13.421693 20428 raft_consensus.cc:2454] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbe0d98735c34b1da7c4a90c93485537 in term 1.
I20260619 13:05:13.422142 20491 leader_election.cc:302] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17cb4ad158de4ad6b15e5765894451ef, 25b5b6ee85e44ac38ed6866d3689849e, cbe0d98735c34b1da7c4a90c93485537; no voters:
I20260619 13:05:13.422513 20627 raft_consensus.cc:2781] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:13.422955 20627 raft_consensus.cc:686] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 1 LEADER]: Becoming Leader. State: Replica: cbe0d98735c34b1da7c4a90c93485537, State: Running, Role: LEADER
I20260619 13:05:13.423524 20627 consensus_queue.cc:237] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } }
I20260619 13:05:13.428223 20155 catalog_manager.cc:5495] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbe0d98735c34b1da7c4a90c93485537 (127.1.87.67). New cstate: current_term: 1 leader_uuid: "cbe0d98735c34b1da7c4a90c93485537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbe0d98735c34b1da7c4a90c93485537" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 35841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 } health_report { overall_health: UNKNOWN } } }
I20260619 13:05:13.448299 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.442734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.1.87.67:35841)
0619 13:05:13.442903 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.442907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.443009 (+ 102us) server_negotiation.cc:407] Connection header received
0619 13:05:13.443156 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.443160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.443238 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.443438 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:05:13.443997 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.444701 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.446155 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.446555 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.446656 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.447159 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:05:13.447729 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:05:13.447854 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.448075 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.448111 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:05:13.457069 20428 raft_consensus.cc:1261] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [term 1 FOLLOWER]: Refusing update from remote peer cbe0d98735c34b1da7c4a90c93485537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:13.457099 20297 raft_consensus.cc:1261] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [term 1 FOLLOWER]: Refusing update from remote peer cbe0d98735c34b1da7c4a90c93485537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:13.458130 20627 consensus_queue.cc:1035] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17cb4ad158de4ad6b15e5765894451ef" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:13.458482 20631 consensus_queue.cc:1035] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5b6ee85e44ac38ed6866d3689849e" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:13.472436 20634 mvcc.cc:205] Tried to move back new op lower bound from 7298557187909242880 to 7298557187786911744. Current Snapshot: MvccSnapshot[applied={T|T < 7298557187909242880}]
I20260619 13:05:13.489305 20156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:13.489605 20156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:13.493366 20156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e691a271ad6545419211a6728daf842e: Sending DeleteTablet for 3 replicas of tablet a9d020b0a03f433d8dcbc25742898ed6
I20260619 13:05:13.494567 20277 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d020b0a03f433d8dcbc25742898ed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:13 UTC) from {username='slave'} at 127.0.0.1:52824
I20260619 13:05:13.494899 20538 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d020b0a03f433d8dcbc25742898ed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:13 UTC) from {username='slave'} at 127.0.0.1:56414
I20260619 13:05:13.495285 20408 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d020b0a03f433d8dcbc25742898ed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:13 UTC) from {username='slave'} at 127.0.0.1:57894
I20260619 13:05:13.496990 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 20215
I20260619 13:05:13.506088 20645 tablet_replica.cc:330] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: stopping tablet replica
I20260619 13:05:13.506804 20645 raft_consensus.cc:2229] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:05:13.507752 20645 raft_consensus.cc:2258] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:13.508512 20644 tablet_replica.cc:330] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: stopping tablet replica
I20260619 13:05:13.509006 20644 raft_consensus.cc:2229] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:13.509615 20644 raft_consensus.cc:2258] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:13.511792 20645 ts_tablet_manager.cc:1905] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:13.512174 20644 ts_tablet_manager.cc:1905] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:13.519344 20645 ts_tablet_manager.cc:1918] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:13.519635 20645 log.cc:1196] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/a9d020b0a03f433d8dcbc25742898ed6
I20260619 13:05:13.520884 20644 ts_tablet_manager.cc:1918] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:13.521087 20644 log.cc:1196] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/a9d020b0a03f433d8dcbc25742898ed6
I20260619 13:05:13.521713 20645 ts_tablet_manager.cc:1939] T a9d020b0a03f433d8dcbc25742898ed6 P cbe0d98735c34b1da7c4a90c93485537: Deleting consensus metadata
I20260619 13:05:13.521723 20644 ts_tablet_manager.cc:1939] T a9d020b0a03f433d8dcbc25742898ed6 P 17cb4ad158de4ad6b15e5765894451ef: Deleting consensus metadata
I20260619 13:05:13.523726 20142 catalog_manager.cc:4841] TS cbe0d98735c34b1da7c4a90c93485537 (127.1.87.67:35841): tablet a9d020b0a03f433d8dcbc25742898ed6 (table test-table [id=d65d4b5ee39e4271a77f7771353a75f6]) successfully deleted
I20260619 13:05:13.523778 20140 catalog_manager.cc:4841] TS 17cb4ad158de4ad6b15e5765894451ef (127.1.87.65:35297): tablet a9d020b0a03f433d8dcbc25742898ed6 (table test-table [id=d65d4b5ee39e4271a77f7771353a75f6]) successfully deleted
I20260619 13:05:13.525408 20646 tablet_replica.cc:330] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: stopping tablet replica
I20260619 13:05:13.525823 20646 raft_consensus.cc:2229] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:13.526321 20646 raft_consensus.cc:2258] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:13.528338 20646 ts_tablet_manager.cc:1905] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:13.534053 20646 ts_tablet_manager.cc:1918] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:13.534240 20646 log.cc:1196] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/a9d020b0a03f433d8dcbc25742898ed6
I20260619 13:05:13.534904 20646 ts_tablet_manager.cc:1939] T a9d020b0a03f433d8dcbc25742898ed6 P 25b5b6ee85e44ac38ed6866d3689849e: Deleting consensus metadata
I20260619 13:05:13.535303 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.498677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.1.87.65:35297)
0619 13:05:13.498920 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.498931 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.523884 (+ 24953us) server_negotiation.cc:407] Connection header received
0619 13:05:13.524261 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.524267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.524371 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.524489 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:13.525128 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.526236 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.528694 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.529328 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.529455 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.533416 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:13.533428 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:13.533445 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:13.534320 (+ 875us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:13.534387 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:13.534416 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:13.534535 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.534742 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.534917 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 13:05:13.536047 20337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:13.537452 20141 catalog_manager.cc:4841] TS 25b5b6ee85e44ac38ed6866d3689849e (127.1.87.66:40999): tablet a9d020b0a03f433d8dcbc25742898ed6 (table test-table [id=d65d4b5ee39e4271a77f7771353a75f6]) successfully deleted
I20260619 13:05:13.688506 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 20215
I20260619 13:05:13.710722 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 20346
I20260619 13:05:13.720600 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.712384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.1.87.66:40999)
0619 13:05:13.712660 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.712666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.712689 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:05:13.712755 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.712762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.712885 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.713003 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:13.713470 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.714648 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.716754 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.717457 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.717600 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.718821 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:13.718833 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:13.718855 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:13.719686 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:13.719748 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:13.719780 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:13.719899 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.720060 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.720176 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260619 13:05:13.721453 20468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:13.828657 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 20346
I20260619 13:05:13.845889 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 20476
I20260619 13:05:13.854207 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:13.846876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.1.87.67:35841)
0619 13:05:13.847101 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:13.847107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:13.847129 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:05:13.847215 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:13.847221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:13.847377 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:13.847502 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:13.847933 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.849132 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.850566 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:13.851246 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:13.851445 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:13.852498 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:13.852509 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:13.852528 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:13.853346 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:13.853412 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:13.853437 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:13.853542 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:13.853680 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 13:05:13.853780 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:05:13.855232 20598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:13.976167 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 20476
I20260619 13:05:13.991919 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 20125
I20260619 13:05:13.993031 20186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:14.100607 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 20125
2026-06-19T13:05:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2906 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:14 dist-test-slave-tpsm krb5kdc[20654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:14 dist-test-slave-tpsm krb5kdc[20654](info): set up 2 sockets
Jun 19 13:05:14 dist-test-slave-tpsm krb5kdc[20654](info): commencing operation
krb5kdc: starting...
W20260619 13:05:15.207643 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:15 dist-test-slave-tpsm krb5kdc[20654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:15.456744 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37289
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:37289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:15.814193 20667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:15.814570 20667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:15.814733 20667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:15.822728 20667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:15.823133 20667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:15.823364 20667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:15.823484 20667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:15.823589 20667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:15.834893 20667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:37289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:37289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:15.836418 20667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:15.837788 20667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:15.849661 20672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:15.850015 20673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:15.850723 20675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:15.851302 20667 server_base.cc:1034] running on GCE node
I20260619 13:05:15.852174 20667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:15.853572 20667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:15.854815 20667 hybrid_clock.cc:648] HybridClock initialized: now 1781874315854774 us; error 55 us; skew 500 ppm
I20260619 13:05:15.855180 20667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:15 dist-test-slave-tpsm krb5kdc[20654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:15.861568 20667 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:15.863425 20667 webserver.cc:460] Webserver started at http://127.1.87.126:40521/ using document root <none> and password file <none>
I20260619 13:05:15.864135 20667 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:15.864289 20667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:15.864603 20667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:15.867645 20667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "3da8731629de48669818f066378e35bc"
format_stamp: "Formatted at 2026-06-19 13:05:15 on dist-test-slave-tpsm"
server_key: "b3483cadc0ca95c5a2932fd91c626c34"
server_key_iv: "be7739072c9ec98d533a84dc5cc5a989"
server_key_version: "encryptionkey@0"
I20260619 13:05:15.868398 20667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "3da8731629de48669818f066378e35bc"
format_stamp: "Formatted at 2026-06-19 13:05:15 on dist-test-slave-tpsm"
server_key: "b3483cadc0ca95c5a2932fd91c626c34"
server_key_iv: "be7739072c9ec98d533a84dc5cc5a989"
server_key_version: "encryptionkey@0"
I20260619 13:05:15.873723 20667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:05:15.877321 20682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:15.878584 20667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:05:15.878773 20667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "3da8731629de48669818f066378e35bc"
format_stamp: "Formatted at 2026-06-19 13:05:15 on dist-test-slave-tpsm"
server_key: "b3483cadc0ca95c5a2932fd91c626c34"
server_key_iv: "be7739072c9ec98d533a84dc5cc5a989"
server_key_version: "encryptionkey@0"
I20260619 13:05:15.878968 20667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:15.903805 20667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:15.908077 20667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:15.908363 20667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:15.926148 20667 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:37289
I20260619 13:05:15.926188 20733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:37289 every 8 connection(s)
I20260619 13:05:15.927484 20667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:15.931119 20734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:15.933089 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 20667
I20260619 13:05:15.933295 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:15.933758 1373 external_mini_cluster.cc:1428] Setting key 99621687eae0bfef88b905f33648461e
I20260619 13:05:15.942866 20734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc: Bootstrap starting.
I20260619 13:05:15.946715 20734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:15.948104 20734 log.cc:824] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:05:15 dist-test-slave-tpsm krb5kdc[20654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:15.951712 20734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc: No bootstrap required, opened a new log
I20260619 13:05:15.958565 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:15.935648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.1.87.126:37289)
0619 13:05:15.936468 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:15.936480 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:15.936518 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:05:15.938132 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:15.938153 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:15.938506 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:15.938799 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:15.940541 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:15.941869 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:15.943674 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:15.944308 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:15.944456 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:15.949902 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:15.949913 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:15.949931 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:15.949986 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:15.953933 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:15.954737 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:15.954746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:15.954754 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:15.954960 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:15.955361 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:15.955366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:15.955369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:15.956084 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:15.956377 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:15.956724 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 13:05:15.957181 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":383,"threads_started":1}
I20260619 13:05:15.962020 20734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8731629de48669818f066378e35bc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37289 } }
I20260619 13:05:15.962340 20734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:15.962415 20734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da8731629de48669818f066378e35bc, State: Initialized, Role: FOLLOWER
I20260619 13:05:15.962872 20734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8731629de48669818f066378e35bc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37289 } }
I20260619 13:05:15.963068 20734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:15.963160 20734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:15.963287 20734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:15.965660 20734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8731629de48669818f066378e35bc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37289 } }
I20260619 13:05:15.966154 20734 leader_election.cc:302] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da8731629de48669818f066378e35bc; no voters:
I20260619 13:05:15.966660 20734 leader_election.cc:288] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:15.966818 20739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:15.967523 20739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [term 1 LEADER]: Becoming Leader. State: Replica: 3da8731629de48669818f066378e35bc, State: Running, Role: LEADER
I20260619 13:05:15.968047 20739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8731629de48669818f066378e35bc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37289 } }
I20260619 13:05:15.968686 20734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:15.970916 20741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da8731629de48669818f066378e35bc. Latest consensus state: current_term: 1 leader_uuid: "3da8731629de48669818f066378e35bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8731629de48669818f066378e35bc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37289 } } }
I20260619 13:05:15.971104 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da8731629de48669818f066378e35bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da8731629de48669818f066378e35bc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 37289 } } }
I20260619 13:05:15.971413 20741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:15.971468 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:15.972458 20747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:15.977250 20747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:15.986366 20747 catalog_manager.cc:1347] Generated new cluster ID: ce9364b20f6b4a99b36dcbcf9dafe039
I20260619 13:05:15.986510 20747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:15.999714 20747 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:16.001567 20747 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:16.012392 20747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc: Generated new TSK 0
I20260619 13:05:16.013476 20747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.87.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
Entry for principal kudu/127.1.87.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab.
I20260619 13:05:16.075941 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37289
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:16.413369 20760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:16.413714 20760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:16.413894 20760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:16.421176 20760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:16.421346 20760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:16.421545 20760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:05:16.431414 20760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.65.keytab
--principal=kudu/127.1.87.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:16.432786 20760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:16.434090 20760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:16.447849 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:16.447891 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:16.447778 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:16.448740 20760 server_base.cc:1034] running on GCE node
I20260619 13:05:16.449456 20760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:16.450505 20760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:16.451694 20760 hybrid_clock.cc:648] HybridClock initialized: now 1781874316451639 us; error 51 us; skew 500 ppm
I20260619 13:05:16.452049 20760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:16 dist-test-slave-tpsm krb5kdc[20654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:16.455291 20760 init.cc:377] Logged in from keytab as kudu/127.1.87.65@KRBTEST.COM (short username kudu)
I20260619 13:05:16.456741 20760 webserver.cc:460] Webserver started at http://127.1.87.65:35313/ using document root <none> and password file <none>
I20260619 13:05:16.457430 20760 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:16.457528 20760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:16.457877 20760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:16.461148 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "b18423fbc12b4d6f9175d9075fa950a9"
format_stamp: "Formatted at 2026-06-19 13:05:16 on dist-test-slave-tpsm"
server_key: "63afeee5058833df6d3e35c0ad2d5dfe"
server_key_iv: "616b1ff21ce22b5a16439752d0915302"
server_key_version: "encryptionkey@0"
I20260619 13:05:16.461941 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "b18423fbc12b4d6f9175d9075fa950a9"
format_stamp: "Formatted at 2026-06-19 13:05:16 on dist-test-slave-tpsm"
server_key: "63afeee5058833df6d3e35c0ad2d5dfe"
server_key_iv: "616b1ff21ce22b5a16439752d0915302"
server_key_version: "encryptionkey@0"
I20260619 13:05:16.467401 20760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 13:05:16.472687 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:16.474030 20760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260619 13:05:16.474220 20760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "b18423fbc12b4d6f9175d9075fa950a9"
format_stamp: "Formatted at 2026-06-19 13:05:16 on dist-test-slave-tpsm"
server_key: "63afeee5058833df6d3e35c0ad2d5dfe"
server_key_iv: "616b1ff21ce22b5a16439752d0915302"
server_key_version: "encryptionkey@0"
I20260619 13:05:16.474427 20760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:16.490921 20760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:16.495440 20760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:16.495733 20760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:16.496706 20760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:16.498008 20760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:16.498116 20760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:16.498272 20760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:16.498347 20760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:16.533623 20760 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:37397
I20260619 13:05:16.533676 20887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:37397 every 8 connection(s)
I20260619 13:05:16.534997 20760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:05:16.543668 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 20760
I20260619 13:05:16.543865 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:05:16.544376 1373 external_mini_cluster.cc:1428] Setting key 4985c4cf2fa219f547141fea870777d4
Jun 19 13:05:16 dist-test-slave-tpsm krb5kdc[20654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.65@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:16.558557 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:16.537569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:35201 (local address 127.1.87.126:37289)
0619 13:05:16.538577 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:16.538583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:16.539447 (+ 864us) server_negotiation.cc:407] Connection header received
0619 13:05:16.541209 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:16.541213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:16.541291 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:16.541417 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:16.543303 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:16.544483 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:16.546522 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:16.547305 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:16.547681 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:16.551478 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:16.551487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:16.551495 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:16.551544 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:16.555197 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:16.556137 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:16.556142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:16.556144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:16.556259 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:16.556794 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:16.556797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:16.556799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:16.557052 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:16.557159 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:16.558124 (+ 965us) server_negotiation.cc:299] Negotiation successful
0619 13:05:16.558298 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":214,"threads_started":1}
I20260619 13:05:16.560230 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:16.538330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37289 (local address 127.1.87.65:35201)
0619 13:05:16.539189 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:16.539243 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:16.540951 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:16.541927 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:16.541938 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:16.542498 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:16.543058 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:16.543076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:16.544666 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:16.544676 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:16.546258 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:16.546270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:16.547759 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:16.547764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:16.547954 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:16.548696 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:16.548723 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:16.551221 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:16.555429 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:16.555437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:16.555451 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:16.555989 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:16.556377 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:16.556385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:16.556390 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:16.556670 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:16.557188 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:16.557200 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:16.557694 (+ 494us) client_negotiation.cc:769] Sending connection context
0619 13:05:16.558025 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 13:05:16.558638 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":375,"threads_started":1}
I20260619 13:05:16.561877 20888 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37289
I20260619 13:05:16.562191 20888 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:16.562914 20888 heartbeater.cc:511] Master 127.1.87.126:37289 requested a full tablet report, sending...
I20260619 13:05:16.565059 20699 ts_manager.cc:194] Registered new tserver with Master: b18423fbc12b4d6f9175d9075fa950a9 (127.1.87.65:37397)
I20260619 13:05:16.566501 20699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.65@KRBTEST.COM'} at 127.1.87.65:35201
WARNING: no policy specified for kudu/127.1.87.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
Entry for principal kudu/127.1.87.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab.
I20260619 13:05:16.602746 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37289
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:16.955001 20894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:16.955399 20894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:16.955576 20894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:16.963609 20894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:16.963792 20894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:16.963972 20894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:05:16.975476 20894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.66.keytab
--principal=kudu/127.1.87.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:16.976792 20894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:16.978128 20894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:16.991897 20899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:16.991842 20900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:16.991873 20902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:16.992612 20894 server_base.cc:1034] running on GCE node
I20260619 13:05:16.993260 20894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:16.994388 20894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:16.995623 20894 hybrid_clock.cc:648] HybridClock initialized: now 1781874316995568 us; error 47 us; skew 500 ppm
I20260619 13:05:16.995987 20894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:16 dist-test-slave-tpsm krb5kdc[20654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:16.999655 20894 init.cc:377] Logged in from keytab as kudu/127.1.87.66@KRBTEST.COM (short username kudu)
I20260619 13:05:17.001170 20894 webserver.cc:460] Webserver started at http://127.1.87.66:39527/ using document root <none> and password file <none>
I20260619 13:05:17.001859 20894 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:17.001959 20894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:17.002230 20894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:17.005299 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "1b072f0164624c9e8454025950f9d5fe"
format_stamp: "Formatted at 2026-06-19 13:05:16 on dist-test-slave-tpsm"
server_key: "a7682cca40f64b99935322f59f81d802"
server_key_iv: "b682b6a1911febb7eebaac3ab9cb29f2"
server_key_version: "encryptionkey@0"
I20260619 13:05:17.006158 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "1b072f0164624c9e8454025950f9d5fe"
format_stamp: "Formatted at 2026-06-19 13:05:16 on dist-test-slave-tpsm"
server_key: "a7682cca40f64b99935322f59f81d802"
server_key_iv: "b682b6a1911febb7eebaac3ab9cb29f2"
server_key_version: "encryptionkey@0"
I20260619 13:05:17.011960 20894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:05:17.015666 20909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:17.017567 20894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:05:17.017807 20894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "1b072f0164624c9e8454025950f9d5fe"
format_stamp: "Formatted at 2026-06-19 13:05:16 on dist-test-slave-tpsm"
server_key: "a7682cca40f64b99935322f59f81d802"
server_key_iv: "b682b6a1911febb7eebaac3ab9cb29f2"
server_key_version: "encryptionkey@0"
I20260619 13:05:17.018026 20894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:17.043619 20894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:17.050524 20894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:17.050922 20894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:17.052096 20894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:17.053833 20894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:17.053965 20894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:17.054121 20894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:17.054203 20894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:17.092183 20894 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:46087
I20260619 13:05:17.092245 21021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:46087 every 8 connection(s)
I20260619 13:05:17.093521 20894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:05:17.101559 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 20894
I20260619 13:05:17.101742 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:05:17.102202 1373 external_mini_cluster.cc:1428] Setting key 8d4206e06adc61b3b97908dfb5abf228
Jun 19 13:05:17 dist-test-slave-tpsm krb5kdc[20654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.66@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:17.117506 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.096179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:40951 (local address 127.1.87.126:37289)
0619 13:05:17.096811 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:17.096821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:17.097695 (+ 874us) server_negotiation.cc:407] Connection header received
0619 13:05:17.099914 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:17.099920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:17.100013 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:17.100148 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:17.101913 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.102974 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.104879 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.105314 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.105418 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.109972 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:17.109981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:17.109988 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:17.110035 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:17.113820 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:17.114846 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:17.114860 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:17.114863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:17.115016 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:17.115689 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:17.115694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:17.115699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:17.115982 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:17.116160 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:17.116894 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 13:05:17.117129 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":407,"threads_started":1}
I20260619 13:05:17.118657 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.096773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37289 (local address 127.1.87.66:40951)
0619 13:05:17.097519 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:17.097564 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:17.099628 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:17.100388 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:17.100395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:17.100913 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:17.101694 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.101723 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.103150 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.103156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.104709 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.104724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.105548 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.105557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.105887 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.106733 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:17.106770 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:17.109695 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:17.114010 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:17.114018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:17.114032 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:17.114666 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:17.115163 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:17.115170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:17.115174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:17.115555 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:17.116130 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:17.116139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:17.116499 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 13:05:17.116788 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 13:05:17.117329 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":280,"threads_started":1}
I20260619 13:05:17.120226 21022 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37289
I20260619 13:05:17.120563 21022 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:17.121429 21022 heartbeater.cc:511] Master 127.1.87.126:37289 requested a full tablet report, sending...
I20260619 13:05:17.123135 20699 ts_manager.cc:194] Registered new tserver with Master: 1b072f0164624c9e8454025950f9d5fe (127.1.87.66:46087)
I20260619 13:05:17.124104 20699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.66@KRBTEST.COM'} at 127.1.87.66:40951
WARNING: no policy specified for kudu/127.1.87.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
Entry for principal kudu/127.1.87.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab.
I20260619 13:05:17.162773 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37289
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:17.510777 21028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:17.511140 21028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:17.511360 21028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:17.519578 21028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:17.519780 21028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:17.519979 21028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:05:17.531796 21028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.67.keytab
--principal=kudu/127.1.87.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:37289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:17.533281 21028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:17.534564 21028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:17.549808 21033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:17.549847 21036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:17.549880 21034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:17.550577 21028 server_base.cc:1034] running on GCE node
I20260619 13:05:17.551457 21028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:17.552517 21028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:17.553745 21028 hybrid_clock.cc:648] HybridClock initialized: now 1781874317553700 us; error 39 us; skew 500 ppm
I20260619 13:05:17.554102 21028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:17 dist-test-slave-tpsm krb5kdc[20654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:17.558902 21028 init.cc:377] Logged in from keytab as kudu/127.1.87.67@KRBTEST.COM (short username kudu)
I20260619 13:05:17.561359 21028 webserver.cc:460] Webserver started at http://127.1.87.67:43045/ using document root <none> and password file <none>
I20260619 13:05:17.562311 21028 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:17.562573 21028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:17.563038 21028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:17.566416 21028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "61dec28b3dad4147be9c5613e8110ae9"
format_stamp: "Formatted at 2026-06-19 13:05:17 on dist-test-slave-tpsm"
server_key: "144088a7cec5697f8f652b1d74e0515f"
server_key_iv: "66984e3ca9c3d8c6c6efec332b795ea2"
server_key_version: "encryptionkey@0"
I20260619 13:05:17.567276 21028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "61dec28b3dad4147be9c5613e8110ae9"
format_stamp: "Formatted at 2026-06-19 13:05:17 on dist-test-slave-tpsm"
server_key: "144088a7cec5697f8f652b1d74e0515f"
server_key_iv: "66984e3ca9c3d8c6c6efec332b795ea2"
server_key_version: "encryptionkey@0"
I20260619 13:05:17.569752 20888 heartbeater.cc:503] Master 127.1.87.126:37289 was elected leader, sending a full tablet report...
I20260619 13:05:17.573680 21028 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 13:05:17.577502 21043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:17.579128 21028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:05:17.579355 21028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "61dec28b3dad4147be9c5613e8110ae9"
format_stamp: "Formatted at 2026-06-19 13:05:17 on dist-test-slave-tpsm"
server_key: "144088a7cec5697f8f652b1d74e0515f"
server_key_iv: "66984e3ca9c3d8c6c6efec332b795ea2"
server_key_version: "encryptionkey@0"
I20260619 13:05:17.579675 21028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:17.600167 21028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:17.605129 21028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:17.605543 21028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:17.606808 21028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:17.609011 21028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:17.609108 21028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:17.609295 21028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:17.609476 21028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:17.648880 21028 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:34585
I20260619 13:05:17.648916 21155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:34585 every 8 connection(s)
I20260619 13:05:17.650219 21028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:05:17.650712 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 21028
I20260619 13:05:17.650895 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:05:17.651515 1373 external_mini_cluster.cc:1428] Setting key 3e6aa28de4ef4355a54f01375eca7b75
Jun 19 13:05:17 dist-test-slave-tpsm krb5kdc[20654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.67@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:17.671633 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.652890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:56177 (local address 127.1.87.126:37289)
0619 13:05:17.653688 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:17.653696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:17.655284 (+ 1588us) server_negotiation.cc:407] Connection header received
0619 13:05:17.656490 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:17.656495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:17.656564 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:17.656664 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:17.658057 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.658876 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.660488 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.660895 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.661000 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.664843 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:17.664847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:17.664850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:17.664883 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:17.668597 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:17.669315 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:17.669319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:17.669321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:17.669483 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:17.669933 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:17.669937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:17.669939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:17.670259 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:17.670396 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:17.671090 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 13:05:17.671266 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":319,"threads_started":1}
I20260619 13:05:17.672595 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.653441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:37289 (local address 127.1.87.67:56177)
0619 13:05:17.654312 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:17.654354 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:17.656220 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:17.656868 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:17.656876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:17.657360 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:17.657908 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.657924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.659031 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.659036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.660339 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.660352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.661099 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.661105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.661295 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.662208 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:17.662232 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:17.664638 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:17.668729 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:17.668736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:17.668749 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:17.669188 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:17.669598 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:17.669602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:17.669604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:17.669785 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:17.670395 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:17.670402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:17.670734 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 13:05:17.670967 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 13:05:17.671441 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":361,"threads_started":1}
I20260619 13:05:17.674081 21156 heartbeater.cc:347] Connected to a master server at 127.1.87.126:37289
I20260619 13:05:17.674374 21156 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:17.675082 21156 heartbeater.cc:511] Master 127.1.87.126:37289 requested a full tablet report, sending...
I20260619 13:05:17.676568 20699 ts_manager.cc:194] Registered new tserver with Master: 61dec28b3dad4147be9c5613e8110ae9 (127.1.87.67:34585)
I20260619 13:05:17.677405 20699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.87.67@KRBTEST.COM'} at 127.1.87.67:56177
I20260619 13:05:17.681677 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:05:17.701088 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.686060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.1.87.126:37289)
0619 13:05:17.686300 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:17.686309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:17.686684 (+ 375us) server_negotiation.cc:407] Connection header received
0619 13:05:17.686907 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:17.686911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:17.686988 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:17.687099 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:17.688538 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.689254 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.690539 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.690933 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.691079 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.694123 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:17.694130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:17.694133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:17.694182 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:17.697266 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:17.698157 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:17.698173 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:17.698181 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:17.698319 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:17.699122 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:17.699129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:17.699132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:17.699599 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:17.699893 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:17.700270 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 13:05:17.700685 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:05:17.705498 20699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:05:17.713824 20699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:05:17.740116 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.728925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.1.87.65:37397)
0619 13:05:17.729766 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:17.729777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:17.730355 (+ 578us) server_negotiation.cc:407] Connection header received
0619 13:05:17.730653 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:17.730664 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:17.730930 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:17.732045 (+ 1115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:17.732312 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.734249 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.737087 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.738499 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.738683 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.738927 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:17.739234 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 13:05:17.739554 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":697,"threads_started":1}
I20260619 13:05:17.740104 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.728782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:37397 (local address 127.0.0.1:44258)
0619 13:05:17.730211 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:17.730244 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:17.730450 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:17.731157 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:17.731166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:17.731205 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:17.731865 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.731879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.734412 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.734417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.736856 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.736871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.738751 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.738758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.738972 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.739026 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 13:05:17.739194 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 13:05:17.739414 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1191,"thread_start_us":445,"threads_started":1}
I20260619 13:05:17.741621 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.729042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:46087 (local address 127.0.0.1:60534)
0619 13:05:17.730962 (+ 1920us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:17.730981 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:17.731101 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:17.732279 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:17.732284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:17.732310 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:17.733585 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.733605 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.735911 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.735917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.738104 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.738119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.740739 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.740748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.740926 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.740956 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:05:17.741064 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:05:17.741233 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"thread_start_us":1669,"threads_started":1}
I20260619 13:05:17.742380 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.729632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.1.87.66:46087)
0619 13:05:17.730223 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:17.730234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:17.731218 (+ 984us) server_negotiation.cc:407] Connection header received
0619 13:05:17.731466 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:17.731564 (+ 98us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:17.731818 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:17.732012 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:17.734130 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.735751 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.738295 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.739961 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.740111 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.740314 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:17.741871 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0619 13:05:17.742045 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":295,"threads_started":1}
I20260619 13:05:17.743814 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.731044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:34585 (local address 127.0.0.1:53726)
0619 13:05:17.734248 (+ 3204us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:17.734271 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:17.734567 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:17.736045 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:17.736050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:17.736083 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:17.736591 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.736605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.738177 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.738185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.741170 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.741184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.743069 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.743073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:17.743187 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.743209 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:05:17.743317 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:05:17.743473 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3050,"thread_start_us":2892,"threads_started":1}
I20260619 13:05:17.744614 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.731449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.1.87.67:34585)
0619 13:05:17.735442 (+ 3993us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:17.735448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:17.735475 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:05:17.735551 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:17.735561 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:17.735789 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:17.736799 (+ 1010us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:17.737033 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.738020 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.741359 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.742662 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.742795 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:17.742966 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:17.744086 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0619 13:05:17.744274 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3802,"thread_start_us":3628,"threads_started":1}
I20260619 13:05:17.744596 20823 tablet_service.cc:1467] Processing CreateTablet for tablet c445ceb344b84766bda0f678e4977e49 (DEFAULT_TABLE table=test-table [id=31191fbd93b3461e9d819b60144f6f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:17.746577 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c445ceb344b84766bda0f678e4977e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:17.746546 20957 tablet_service.cc:1467] Processing CreateTablet for tablet c445ceb344b84766bda0f678e4977e49 (DEFAULT_TABLE table=test-table [id=31191fbd93b3461e9d819b60144f6f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:17.747903 20957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c445ceb344b84766bda0f678e4977e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:17.747905 21091 tablet_service.cc:1467] Processing CreateTablet for tablet c445ceb344b84766bda0f678e4977e49 (DEFAULT_TABLE table=test-table [id=31191fbd93b3461e9d819b60144f6f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:17.749204 21091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c445ceb344b84766bda0f678e4977e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:17.759294 21175 tablet_bootstrap.cc:492] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: Bootstrap starting.
I20260619 13:05:17.763581 21175 tablet_bootstrap.cc:654] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:17.764063 21176 tablet_bootstrap.cc:492] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: Bootstrap starting.
I20260619 13:05:17.765161 21175 log.cc:824] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:17.765473 21177 tablet_bootstrap.cc:492] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: Bootstrap starting.
I20260619 13:05:17.768184 21177 tablet_bootstrap.cc:654] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:17.768251 21176 tablet_bootstrap.cc:654] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:17.769816 21177 log.cc:824] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:17.770851 21176 log.cc:824] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:17.773485 21175 tablet_bootstrap.cc:492] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: No bootstrap required, opened a new log
I20260619 13:05:17.773813 21175 ts_tablet_manager.cc:1397] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260619 13:05:17.773831 21177 tablet_bootstrap.cc:492] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: No bootstrap required, opened a new log
I20260619 13:05:17.774096 21177 ts_tablet_manager.cc:1397] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 13:05:17.774196 21176 tablet_bootstrap.cc:492] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: No bootstrap required, opened a new log
I20260619 13:05:17.774446 21176 ts_tablet_manager.cc:1397] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260619 13:05:17.783622 21175 raft_consensus.cc:348] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } }
I20260619 13:05:17.784104 21175 raft_consensus.cc:374] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:17.784263 21175 raft_consensus.cc:729] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18423fbc12b4d6f9175d9075fa950a9, State: Initialized, Role: FOLLOWER
I20260619 13:05:17.784977 21175 consensus_queue.cc:260] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } }
I20260619 13:05:17.786180 21177 raft_consensus.cc:348] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } }
I20260619 13:05:17.786682 21177 raft_consensus.cc:374] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:17.786782 21177 raft_consensus.cc:729] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61dec28b3dad4147be9c5613e8110ae9, State: Initialized, Role: FOLLOWER
I20260619 13:05:17.786629 21176 raft_consensus.cc:348] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } }
I20260619 13:05:17.787027 21175 ts_tablet_manager.cc:1428] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260619 13:05:17.787070 21176 raft_consensus.cc:374] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:17.787175 21176 raft_consensus.cc:729] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b072f0164624c9e8454025950f9d5fe, State: Initialized, Role: FOLLOWER
W20260619 13:05:17.787467 20889 tablet.cc:2369] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:05:17.787851 21176 consensus_queue.cc:260] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } }
I20260619 13:05:17.789038 21022 heartbeater.cc:503] Master 127.1.87.126:37289 was elected leader, sending a full tablet report...
I20260619 13:05:17.789801 21176 ts_tablet_manager.cc:1428] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260619 13:05:17.788640 21177 consensus_queue.cc:260] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } }
I20260619 13:05:17.791570 21156 heartbeater.cc:503] Master 127.1.87.126:37289 was elected leader, sending a full tablet report...
I20260619 13:05:17.792150 21177 ts_tablet_manager.cc:1428] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
W20260619 13:05:17.845779 21023 tablet.cc:2369] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:05:17.902120 21157 tablet.cc:2369] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:05:17.990310 21182 raft_consensus.cc:482] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:05:17.990757 21182 raft_consensus.cc:504] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } }
I20260619 13:05:17.992492 21182 leader_election.cc:288] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b18423fbc12b4d6f9175d9075fa950a9 (127.1.87.65:37397), 61dec28b3dad4147be9c5613e8110ae9 (127.1.87.67:34585)
I20260619 13:05:18.002938 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.993072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:52277 (local address 127.1.87.65:37397)
0619 13:05:17.993314 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:17.993322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:17.994002 (+ 680us) server_negotiation.cc:407] Connection header received
0619 13:05:17.994114 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:17.994120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:17.994205 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:17.994315 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:17.995148 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.996753 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.000271 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.001571 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.001723 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:18.001833 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:18.002528 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 13:05:18.002670 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:05:18.002938 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.993083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:50827 (local address 127.1.87.67:34585)
0619 13:05:17.993433 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:17.993439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:17.993909 (+ 470us) server_negotiation.cc:407] Connection header received
0619 13:05:17.994120 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:17.994126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:17.994205 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:17.994315 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:17.995148 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.996759 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.000279 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.001594 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.001723 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:18.001833 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:18.002528 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 13:05:18.002680 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260619 13:05:18.003456 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.992946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:37397 (local address 127.1.87.66:52277)
0619 13:05:17.993884 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:17.993898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:17.993989 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:17.994355 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:17.994360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:17.994452 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:17.994962 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.994975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.996912 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.996917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:18.000067 (+ 3150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:18.000083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.001921 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.001929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:18.002253 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:18.002368 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 13:05:18.002477 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 13:05:18.002669 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":434,"spinlock_wait_cycles":21888,"thread_start_us":636,"threads_started":1}
I20260619 13:05:18.003495 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:17.992904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:34585 (local address 127.1.87.66:50827)
0619 13:05:17.993788 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:17.993809 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:17.993977 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:17.994355 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:17.994360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:17.994388 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:17.994962 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:17.994975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:17.996912 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:17.996918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:18.000067 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:18.000083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.001923 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.001929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:18.002258 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:18.002368 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 13:05:18.002477 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 13:05:18.002669 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":347,"thread_start_us":396,"threads_started":1}
I20260619 13:05:18.004696 21111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c445ceb344b84766bda0f678e4977e49" candidate_uuid: "1b072f0164624c9e8454025950f9d5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61dec28b3dad4147be9c5613e8110ae9" is_pre_election: true
I20260619 13:05:18.004698 20843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c445ceb344b84766bda0f678e4977e49" candidate_uuid: "1b072f0164624c9e8454025950f9d5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18423fbc12b4d6f9175d9075fa950a9" is_pre_election: true
I20260619 13:05:18.005307 20843 raft_consensus.cc:2454] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b072f0164624c9e8454025950f9d5fe in term 0.
I20260619 13:05:18.005307 21111 raft_consensus.cc:2454] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b072f0164624c9e8454025950f9d5fe in term 0.
I20260619 13:05:18.006095 20912 leader_election.cc:302] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b072f0164624c9e8454025950f9d5fe, 61dec28b3dad4147be9c5613e8110ae9, b18423fbc12b4d6f9175d9075fa950a9; no voters:
I20260619 13:05:18.006495 21182 raft_consensus.cc:2781] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:05:18.006656 21182 raft_consensus.cc:482] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:05:18.006778 21182 raft_consensus.cc:3037] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:18.009146 21182 raft_consensus.cc:504] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } }
I20260619 13:05:18.009847 21182 leader_election.cc:288] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [CANDIDATE]: Term 1 election: Requested vote from peers b18423fbc12b4d6f9175d9075fa950a9 (127.1.87.65:37397), 61dec28b3dad4147be9c5613e8110ae9 (127.1.87.67:34585)
I20260619 13:05:18.010349 21111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c445ceb344b84766bda0f678e4977e49" candidate_uuid: "1b072f0164624c9e8454025950f9d5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61dec28b3dad4147be9c5613e8110ae9"
I20260619 13:05:18.010321 20843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c445ceb344b84766bda0f678e4977e49" candidate_uuid: "1b072f0164624c9e8454025950f9d5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18423fbc12b4d6f9175d9075fa950a9"
I20260619 13:05:18.010555 21111 raft_consensus.cc:3037] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:18.010710 20843 raft_consensus.cc:3037] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:18.012933 21111 raft_consensus.cc:2454] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b072f0164624c9e8454025950f9d5fe in term 1.
I20260619 13:05:18.013089 20843 raft_consensus.cc:2454] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b072f0164624c9e8454025950f9d5fe in term 1.
I20260619 13:05:18.013540 20912 leader_election.cc:302] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b072f0164624c9e8454025950f9d5fe, b18423fbc12b4d6f9175d9075fa950a9; no voters:
I20260619 13:05:18.013828 21182 raft_consensus.cc:2781] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:18.014432 21182 raft_consensus.cc:686] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 1 LEADER]: Becoming Leader. State: Replica: 1b072f0164624c9e8454025950f9d5fe, State: Running, Role: LEADER
I20260619 13:05:18.014947 21182 consensus_queue.cc:237] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } }
I20260619 13:05:18.020048 20697 catalog_manager.cc:5495] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b072f0164624c9e8454025950f9d5fe (127.1.87.66). New cstate: current_term: 1 leader_uuid: "1b072f0164624c9e8454025950f9d5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b072f0164624c9e8454025950f9d5fe" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 46087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 } health_report { overall_health: UNKNOWN } } }
I20260619 13:05:18.039249 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:18.033893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.1.87.66:46087)
0619 13:05:18.034092 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:18.034096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:18.034114 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:05:18.034200 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:18.034203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:18.034277 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:18.034400 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:05:18.034950 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.035749 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.037117 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.037531 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.037625 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:18.038103 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:05:18.038746 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:05:18.038858 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:18.038968 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 13:05:18.039036 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:05:18.047191 21111 raft_consensus.cc:1261] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [term 1 FOLLOWER]: Refusing update from remote peer 1b072f0164624c9e8454025950f9d5fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:18.047191 20843 raft_consensus.cc:1261] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [term 1 FOLLOWER]: Refusing update from remote peer 1b072f0164624c9e8454025950f9d5fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:18.048179 21186 consensus_queue.cc:1035] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "61dec28b3dad4147be9c5613e8110ae9" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:18.048574 21182 consensus_queue.cc:1035] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b18423fbc12b4d6f9175d9075fa950a9" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:18.060717 21189 mvcc.cc:205] Tried to move back new op lower bound from 7298557206711947264 to 7298557206593531904. Current Snapshot: MvccSnapshot[applied={T|T < 7298557206711947264}]
I20260619 13:05:18.078615 20697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:18.079169 20697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:18.084267 20697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3da8731629de48669818f066378e35bc: Sending DeleteTablet for 3 replicas of tablet c445ceb344b84766bda0f678e4977e49
I20260619 13:05:18.085803 20823 tablet_service.cc:1514] Processing DeleteTablet for tablet c445ceb344b84766bda0f678e4977e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:18 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:44258
I20260619 13:05:18.086261 21091 tablet_service.cc:1514] Processing DeleteTablet for tablet c445ceb344b84766bda0f678e4977e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:18 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:53726
I20260619 13:05:18.086624 20957 tablet_service.cc:1514] Processing DeleteTablet for tablet c445ceb344b84766bda0f678e4977e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:18 UTC) from {username='kudu', principal='kudu/127.1.87.126@KRBTEST.COM'} at 127.0.0.1:60534
I20260619 13:05:18.086966 21200 tablet_replica.cc:330] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: stopping tablet replica
I20260619 13:05:18.087488 21200 raft_consensus.cc:2229] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:18.088106 21200 raft_consensus.cc:2258] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:18.088935 21202 tablet_replica.cc:330] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: stopping tablet replica
I20260619 13:05:18.089442 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 20760
I20260619 13:05:18.089412 21202 raft_consensus.cc:2229] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:05:18.090188 21202 raft_consensus.cc:2258] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:18.091037 21200 ts_tablet_manager.cc:1905] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:18.092737 21202 ts_tablet_manager.cc:1905] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:18.100778 21202 ts_tablet_manager.cc:1918] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:18.100888 21200 ts_tablet_manager.cc:1918] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:18.101114 21200 log.cc:1196] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/c445ceb344b84766bda0f678e4977e49
I20260619 13:05:18.101116 21202 log.cc:1196] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/c445ceb344b84766bda0f678e4977e49
I20260619 13:05:18.101805 21200 ts_tablet_manager.cc:1939] T c445ceb344b84766bda0f678e4977e49 P b18423fbc12b4d6f9175d9075fa950a9: Deleting consensus metadata
I20260619 13:05:18.101979 21202 ts_tablet_manager.cc:1939] T c445ceb344b84766bda0f678e4977e49 P 1b072f0164624c9e8454025950f9d5fe: Deleting consensus metadata
I20260619 13:05:18.103780 21201 tablet_replica.cc:330] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: stopping tablet replica
I20260619 13:05:18.104280 21201 raft_consensus.cc:2229] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:18.104282 20686 catalog_manager.cc:4841] TS 1b072f0164624c9e8454025950f9d5fe (127.1.87.66:46087): tablet c445ceb344b84766bda0f678e4977e49 (table test-table [id=31191fbd93b3461e9d819b60144f6f07]) successfully deleted
I20260619 13:05:18.104773 21201 raft_consensus.cc:2258] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:18.106415 20685 catalog_manager.cc:4841] TS b18423fbc12b4d6f9175d9075fa950a9 (127.1.87.65:37397): tablet c445ceb344b84766bda0f678e4977e49 (table test-table [id=31191fbd93b3461e9d819b60144f6f07]) successfully deleted
I20260619 13:05:18.108501 21201 ts_tablet_manager.cc:1905] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:18.116547 21201 ts_tablet_manager.cc:1918] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:18.116755 21201 log.cc:1196] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/c445ceb344b84766bda0f678e4977e49
I20260619 13:05:18.117285 21201 ts_tablet_manager.cc:1939] T c445ceb344b84766bda0f678e4977e49 P 61dec28b3dad4147be9c5613e8110ae9: Deleting consensus metadata
I20260619 13:05:18.119597 20686 catalog_manager.cc:4841] TS 61dec28b3dad4147be9c5613e8110ae9 (127.1.87.67:34585): tablet c445ceb344b84766bda0f678e4977e49 (table test-table [id=31191fbd93b3461e9d819b60144f6f07]) successfully deleted
Jun 19 13:05:18 dist-test-slave-tpsm krb5kdc[20654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.65@KRBTEST.COM
I20260619 13:05:18.129235 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:18.090478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.1.87.65:37397)
0619 13:05:18.090665 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:18.090673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:18.111214 (+ 20541us) server_negotiation.cc:407] Connection header received
0619 13:05:18.111720 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:18.111727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:18.111825 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:18.111911 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:18.113809 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.114960 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.116887 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.117576 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.117672 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:18.122862 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:18.122883 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:18.122903 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:18.122951 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:18.126058 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:18.126822 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:18.126835 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:18.126837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:18.126981 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:18.127390 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:18.127395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:18.127400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:18.127769 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:18.128083 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:18.128634 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 13:05:18.128867 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:05:18.129959 20883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:18.244280 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 20760
I20260619 13:05:18.262944 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 20894
Jun 19 13:05:18 dist-test-slave-tpsm krb5kdc[20654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.66@KRBTEST.COM
I20260619 13:05:18.280349 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:18.263890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.1.87.66:46087)
0619 13:05:18.264067 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:18.264075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:18.264104 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:05:18.264315 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:18.264326 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:18.264412 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:18.264527 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:18.265843 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.266671 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.268098 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.268565 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.268709 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:18.273751 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:18.273765 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:18.273782 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:18.273826 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:18.276975 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:18.277871 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:18.277877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:18.277880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:18.278014 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:18.278528 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:18.278534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:18.278537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:18.279196 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:18.279550 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:18.279859 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 13:05:18.280024 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:05:18.281162 21017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:18.398607 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 20894
I20260619 13:05:18.418926 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21028
Jun 19 13:05:18 dist-test-slave-tpsm krb5kdc[20654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.67@KRBTEST.COM
I20260619 13:05:18.435707 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:18.419946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.1.87.67:34585)
0619 13:05:18.420162 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:18.420169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:18.420190 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:05:18.420268 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:18.420272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:18.420336 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:18.420447 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:18.421717 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.422435 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.423780 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:18.424170 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:18.424314 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:18.429697 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:18.429712 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:18.429737 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:18.429776 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:18.432820 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:18.433590 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:18.433604 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:18.433606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:18.433762 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:18.434146 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:18.434152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:18.434155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:18.434497 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:18.434799 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:18.435117 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 13:05:18.435356 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:05:18.436343 21151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:18.530769 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 21028
I20260619 13:05:18.550498 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 20667
I20260619 13:05:18.551797 20729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:18.668411 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 20667
2026-06-19T13:05:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4569 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T13:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:18Z Disabled control of system clock
I20260619 13:05:18.751065 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:35681
--webserver_interface=127.1.87.126
--webserver_port=0
--builtin_ntp_servers=127.1.87.84:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:35681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:05:19.081439 21211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:19.081759 21211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:19.081844 21211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:19.089254 21211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:19.089432 21211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:19.089476 21211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:19.089529 21211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:19.089577 21211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:19.099675 21211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:35681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:35681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:19.100993 21211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:19.102792 21211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:19.116590 21216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:19.116693 21217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:19.116732 21219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:19.117587 21211 server_base.cc:1034] running on GCE node
I20260619 13:05:19.118461 21211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:19.120154 21211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:19.121505 21211 hybrid_clock.cc:648] HybridClock initialized: now 1781874319121438 us; error 56 us; skew 500 ppm
I20260619 13:05:19.122087 21211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:19.125247 21211 webserver.cc:460] Webserver started at http://127.1.87.126:37297/ using document root <none> and password file <none>
I20260619 13:05:19.126101 21211 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:19.126252 21211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:19.126586 21211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:19.129693 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "973593d19eda4854be2c9abdd83cc094"
format_stamp: "Formatted at 2026-06-19 13:05:19 on dist-test-slave-tpsm"
server_key: "db9adfcfe4ccfb27efbc6567df14d2a0"
server_key_iv: "1df83de8088ad9b6d08a041f6851669c"
server_key_version: "encryptionkey@0"
I20260619 13:05:19.130756 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "973593d19eda4854be2c9abdd83cc094"
format_stamp: "Formatted at 2026-06-19 13:05:19 on dist-test-slave-tpsm"
server_key: "db9adfcfe4ccfb27efbc6567df14d2a0"
server_key_iv: "1df83de8088ad9b6d08a041f6851669c"
server_key_version: "encryptionkey@0"
I20260619 13:05:19.137164 21211 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260619 13:05:19.141651 21225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:19.143412 21211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:05:19.143652 21211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "973593d19eda4854be2c9abdd83cc094"
format_stamp: "Formatted at 2026-06-19 13:05:19 on dist-test-slave-tpsm"
server_key: "db9adfcfe4ccfb27efbc6567df14d2a0"
server_key_iv: "1df83de8088ad9b6d08a041f6851669c"
server_key_version: "encryptionkey@0"
I20260619 13:05:19.143905 21211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:19.159161 21211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:19.160140 21211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:19.160459 21211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:19.181695 21211 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:35681
I20260619 13:05:19.181733 21276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:35681 every 8 connection(s)
I20260619 13:05:19.183107 21211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:19.187042 21277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:19.187539 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 21211
I20260619 13:05:19.187822 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:19.188272 1373 external_mini_cluster.cc:1428] Setting key f1b0f5e5cee6d10dc5964f4df53ef88a
I20260619 13:05:19.201347 21277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094: Bootstrap starting.
I20260619 13:05:19.202579 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:19.190331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.1.87.126:35681)
0619 13:05:19.191402 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:19.191413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:19.191458 (+ 45us) server_negotiation.cc:407] Connection header received
0619 13:05:19.193341 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:19.193362 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:19.193894 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:19.194242 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:19.194849 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:19.196172 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:19.198118 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:19.198834 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:19.198989 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:19.200275 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:19.200282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:19.200295 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:19.200820 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:19.200899 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:19.200919 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:19.201009 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:19.201244 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 13:05:19.201574 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":382,"threads_started":1}
I20260619 13:05:19.204706 21277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:19.205852 21277 log.cc:824] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:19.208532 21277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094: No bootstrap required, opened a new log
I20260619 13:05:19.216588 21277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973593d19eda4854be2c9abdd83cc094" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35681 } }
I20260619 13:05:19.217123 21277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:19.217276 21277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973593d19eda4854be2c9abdd83cc094, State: Initialized, Role: FOLLOWER
I20260619 13:05:19.217959 21277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973593d19eda4854be2c9abdd83cc094" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35681 } }
I20260619 13:05:19.218196 21277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:19.218338 21277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:19.218497 21277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:19.220752 21277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973593d19eda4854be2c9abdd83cc094" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35681 } }
I20260619 13:05:19.221297 21277 leader_election.cc:302] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 973593d19eda4854be2c9abdd83cc094; no voters:
I20260619 13:05:19.221911 21277 leader_election.cc:288] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:19.222065 21282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:19.223135 21282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [term 1 LEADER]: Becoming Leader. State: Replica: 973593d19eda4854be2c9abdd83cc094, State: Running, Role: LEADER
I20260619 13:05:19.223789 21282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973593d19eda4854be2c9abdd83cc094" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35681 } }
I20260619 13:05:19.224153 21277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:19.226789 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "973593d19eda4854be2c9abdd83cc094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973593d19eda4854be2c9abdd83cc094" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35681 } } }
I20260619 13:05:19.226895 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973593d19eda4854be2c9abdd83cc094. Latest consensus state: current_term: 1 leader_uuid: "973593d19eda4854be2c9abdd83cc094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "973593d19eda4854be2c9abdd83cc094" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 35681 } } }
I20260619 13:05:19.227290 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:19.227288 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:19.228921 21291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:19.234107 21291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:19.245713 21291 catalog_manager.cc:1347] Generated new cluster ID: f66611ef3afb45728530b9fb6e93e1b4
I20260619 13:05:19.245859 21291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:19.272246 21291 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:19.273581 21291 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:19.286845 21291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094: Generated new TSK 0
I20260619 13:05:19.287837 21291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:19.292920 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.65:0
--local_ip_for_outbound_sockets=127.1.87.65
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:35681
--builtin_ntp_servers=127.1.87.84:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:05:19.636792 21301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:19.637113 21301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:19.637269 21301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:19.644817 21301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:19.644963 21301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:19.645141 21301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.65
I20260619 13:05:19.654680 21301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.87.65
--webserver_port=0
--tserver_master_addrs=127.1.87.126:35681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.65
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:19.656095 21301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:19.657989 21301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:19.671381 21306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:19.671345 21309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:19.671403 21307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:19.672406 21301 server_base.cc:1034] running on GCE node
I20260619 13:05:19.673015 21301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:19.673993 21301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:19.675242 21301 hybrid_clock.cc:648] HybridClock initialized: now 1781874319675198 us; error 39 us; skew 500 ppm
I20260619 13:05:19.675633 21301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:19.678018 21301 webserver.cc:460] Webserver started at http://127.1.87.65:37483/ using document root <none> and password file <none>
I20260619 13:05:19.678735 21301 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:19.678869 21301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:19.679194 21301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:19.682121 21301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/data/instance:
uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3"
format_stamp: "Formatted at 2026-06-19 13:05:19 on dist-test-slave-tpsm"
server_key: "0c4a9c4a0c507ec548659c423292a1d4"
server_key_iv: "7e5a53168ff43724ff1fb552f6f92d59"
server_key_version: "encryptionkey@0"
I20260619 13:05:19.682930 21301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance:
uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3"
format_stamp: "Formatted at 2026-06-19 13:05:19 on dist-test-slave-tpsm"
server_key: "0c4a9c4a0c507ec548659c423292a1d4"
server_key_iv: "7e5a53168ff43724ff1fb552f6f92d59"
server_key_version: "encryptionkey@0"
I20260619 13:05:19.688187 21301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:05:19.691946 21315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:19.693226 21301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:19.693416 21301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/wal
uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3"
format_stamp: "Formatted at 2026-06-19 13:05:19 on dist-test-slave-tpsm"
server_key: "0c4a9c4a0c507ec548659c423292a1d4"
server_key_iv: "7e5a53168ff43724ff1fb552f6f92d59"
server_key_version: "encryptionkey@0"
I20260619 13:05:19.693646 21301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:19.712491 21301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:19.713366 21301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:19.713665 21301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:19.714684 21301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:19.716365 21301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:19.716495 21301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:19.716645 21301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:19.716743 21301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:19.752740 21301 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.65:42819
I20260619 13:05:19.752833 21427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.65:42819 every 8 connection(s)
I20260619 13:05:19.754123 21301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/data/info.pb
I20260619 13:05:19.762115 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 21301
I20260619 13:05:19.762301 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/wal/instance
I20260619 13:05:19.762779 1373 external_mini_cluster.cc:1428] Setting key 2660b660267a54ef624fb66818b88bfe
I20260619 13:05:19.766150 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.66:0
--local_ip_for_outbound_sockets=127.1.87.66
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:35681
--builtin_ntp_servers=127.1.87.84:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:05:19.768231 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:19.756803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.65:58489 (local address 127.1.87.126:35681)
0619 13:05:19.757243 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:19.757247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:19.758422 (+ 1175us) server_negotiation.cc:407] Connection header received
0619 13:05:19.760327 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:19.760333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:19.760429 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:19.760556 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:19.761600 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:19.762437 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:19.764389 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:19.765159 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:19.765307 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:19.767092 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:19.767102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:19.767110 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:19.767139 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:19.767197 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:19.767226 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:19.767374 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:19.767622 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 13:05:19.767740 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260619 13:05:19.768960 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:19.757356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:35681 (local address 127.1.87.65:58489)
0619 13:05:19.758249 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:19.758283 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:19.760051 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:19.760782 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:19.760791 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:19.760873 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:19.761427 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:19.761443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:19.762591 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:19.762596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:19.764222 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:19.764237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:19.765508 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:19.765519 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:19.765885 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:19.766730 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:05:19.766733 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:19.766782 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:05:19.766864 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:19.767401 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:19.767410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:19.767416 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:05:19.767571 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 13:05:19.767877 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":281,"threads_started":1}
I20260619 13:05:19.770551 21428 heartbeater.cc:347] Connected to a master server at 127.1.87.126:35681
I20260619 13:05:19.770903 21428 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:19.772090 21428 heartbeater.cc:511] Master 127.1.87.126:35681 requested a full tablet report, sending...
I20260619 13:05:19.774281 21242 ts_manager.cc:194] Registered new tserver with Master: 40e8e8cb54cc4a77a8f9b0161b0158c3 (127.1.87.65:42819)
I20260619 13:05:19.776551 21242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.65:58489
W20260619 13:05:20.105396 21432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:20.105690 21432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:20.105791 21432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:20.113261 21432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:20.113379 21432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:20.113488 21432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.66
I20260619 13:05:20.123548 21432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.87.66
--webserver_port=0
--tserver_master_addrs=127.1.87.126:35681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.66
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:20.124902 21432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:20.126724 21432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:20.140331 21437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:20.140350 21438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:20.140234 21440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:20.141624 21432 server_base.cc:1034] running on GCE node
I20260619 13:05:20.142256 21432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:20.143261 21432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:20.144716 21432 hybrid_clock.cc:648] HybridClock initialized: now 1781874320144598 us; error 108 us; skew 500 ppm
I20260619 13:05:20.145174 21432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:20.149351 21432 webserver.cc:460] Webserver started at http://127.1.87.66:38511/ using document root <none> and password file <none>
I20260619 13:05:20.150172 21432 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:20.150313 21432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:20.150657 21432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:20.153759 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/data/instance:
uuid: "fd23e88dc94a479496dbe9559774d085"
format_stamp: "Formatted at 2026-06-19 13:05:20 on dist-test-slave-tpsm"
server_key: "0b25770c8d60ea186df5e576020b77d7"
server_key_iv: "2e91e94690e1a388d22f117f8ba2a0c4"
server_key_version: "encryptionkey@0"
I20260619 13:05:20.154603 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance:
uuid: "fd23e88dc94a479496dbe9559774d085"
format_stamp: "Formatted at 2026-06-19 13:05:20 on dist-test-slave-tpsm"
server_key: "0b25770c8d60ea186df5e576020b77d7"
server_key_iv: "2e91e94690e1a388d22f117f8ba2a0c4"
server_key_version: "encryptionkey@0"
I20260619 13:05:20.160084 21432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:05:20.164148 21446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:20.166077 21432 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 13:05:20.166281 21432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/wal
uuid: "fd23e88dc94a479496dbe9559774d085"
format_stamp: "Formatted at 2026-06-19 13:05:20 on dist-test-slave-tpsm"
server_key: "0b25770c8d60ea186df5e576020b77d7"
server_key_iv: "2e91e94690e1a388d22f117f8ba2a0c4"
server_key_version: "encryptionkey@0"
I20260619 13:05:20.166591 21432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:20.182825 21432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:20.183861 21432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:20.184137 21432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:20.185273 21432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:20.187088 21432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:20.187201 21432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:20.187335 21432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:20.187412 21432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 13:05:20.221583 21432 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.66:39001
I20260619 13:05:20.221659 21558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.66:39001 every 8 connection(s)
I20260619 13:05:20.222872 21432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/data/info.pb
I20260619 13:05:20.225248 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 21432
I20260619 13:05:20.225438 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/wal/instance
I20260619 13:05:20.225971 1373 external_mini_cluster.cc:1428] Setting key 210f5d26a74ac03247dfcf5c28215dfd
I20260619 13:05:20.228739 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.87.67:0
--local_ip_for_outbound_sockets=127.1.87.67
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:35681
--builtin_ntp_servers=127.1.87.84:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:05:20.236860 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.225521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:55229 (local address 127.1.87.126:35681)
0619 13:05:20.225762 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.225768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.227307 (+ 1539us) server_negotiation.cc:407] Connection header received
0619 13:05:20.229311 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.229319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.229411 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.229524 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:20.230535 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.231698 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.233233 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.233962 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.234109 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.235854 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:20.235862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:20.235870 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:20.235898 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:20.235941 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:20.235959 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:20.236062 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.236351 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.236468 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:05:20.238152 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.226095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:35681 (local address 127.1.87.66:55229)
0619 13:05:20.227138 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:20.227189 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:20.229041 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:20.229716 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:20.229724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:20.229805 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:20.230359 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.230375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.231841 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.231845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.233067 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.233079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.234183 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.234187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.234400 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.235346 (+ 946us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:05:20.235353 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:20.235442 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:05:20.235565 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:20.236104 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:20.236116 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:20.236123 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:05:20.236300 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 13:05:20.236651 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":348,"threads_started":1}
I20260619 13:05:20.239717 21559 heartbeater.cc:347] Connected to a master server at 127.1.87.126:35681
I20260619 13:05:20.240036 21559 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:20.240840 21559 heartbeater.cc:511] Master 127.1.87.126:35681 requested a full tablet report, sending...
I20260619 13:05:20.242316 21242 ts_manager.cc:194] Registered new tserver with Master: fd23e88dc94a479496dbe9559774d085 (127.1.87.66:39001)
I20260619 13:05:20.243140 21242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.66:55229
W20260619 13:05:20.573479 21562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:20.573815 21562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:20.573982 21562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:20.581367 21562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:20.581555 21562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:20.581760 21562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.87.67
I20260619 13:05:20.591825 21562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.87.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.87.67
--webserver_port=0
--tserver_master_addrs=127.1.87.126:35681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.87.67
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:20.593182 21562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:20.594969 21562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:20.609115 21568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:20.609210 21567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:20.609153 21570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:20.610061 21562 server_base.cc:1034] running on GCE node
I20260619 13:05:20.610728 21562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:20.611752 21562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:20.613020 21562 hybrid_clock.cc:648] HybridClock initialized: now 1781874320612965 us; error 77 us; skew 500 ppm
I20260619 13:05:20.613435 21562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:05:20.616077 21562 webserver.cc:460] Webserver started at http://127.1.87.67:45629/ using document root <none> and password file <none>
I20260619 13:05:20.616808 21562 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:20.616955 21562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:20.617316 21562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:20.620414 21562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/data/instance:
uuid: "931b8017824e400790efba6d422b2687"
format_stamp: "Formatted at 2026-06-19 13:05:20 on dist-test-slave-tpsm"
server_key: "9df6a0222a04c1e1d318ab630d941216"
server_key_iv: "98598019213d66300a1fafbaaa2c645f"
server_key_version: "encryptionkey@0"
I20260619 13:05:20.621271 21562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance:
uuid: "931b8017824e400790efba6d422b2687"
format_stamp: "Formatted at 2026-06-19 13:05:20 on dist-test-slave-tpsm"
server_key: "9df6a0222a04c1e1d318ab630d941216"
server_key_iv: "98598019213d66300a1fafbaaa2c645f"
server_key_version: "encryptionkey@0"
I20260619 13:05:20.627727 21562 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 13:05:20.631501 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:20.632714 21562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:20.632895 21562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/wal
uuid: "931b8017824e400790efba6d422b2687"
format_stamp: "Formatted at 2026-06-19 13:05:20 on dist-test-slave-tpsm"
server_key: "9df6a0222a04c1e1d318ab630d941216"
server_key_iv: "98598019213d66300a1fafbaaa2c645f"
server_key_version: "encryptionkey@0"
I20260619 13:05:20.633126 21562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:20.661506 21562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:20.662660 21562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:20.662983 21562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:20.664022 21562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:05:20.665480 21562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:05:20.665596 21562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:20.665709 21562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:05:20.665781 21562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:20.698282 21562 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.67:34771
I20260619 13:05:20.698376 21688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.67:34771 every 8 connection(s)
I20260619 13:05:20.699584 21562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/data/info.pb
I20260619 13:05:20.706840 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 21562
I20260619 13:05:20.707053 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/wal/instance
I20260619 13:05:20.707607 1373 external_mini_cluster.cc:1428] Setting key b7dc8a08002eebcbf932814927be383c
I20260619 13:05:20.712258 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.701964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.67:42397 (local address 127.1.87.126:35681)
0619 13:05:20.702143 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.702147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.703490 (+ 1343us) server_negotiation.cc:407] Connection header received
0619 13:05:20.705213 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.705218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.705309 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.705421 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:20.706399 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.707236 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.709008 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.709671 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.709775 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.711361 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:20.711368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:20.711445 (+ 77us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:20.711472 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:20.711505 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:20.711522 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:20.711606 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.711933 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.712039 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:05:20.713178 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.702439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:35681 (local address 127.1.87.67:42397)
0619 13:05:20.703282 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:20.703374 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:20.704967 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:20.705634 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:20.705642 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:20.705721 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:20.706239 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.706254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.707409 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.707413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.708813 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.708829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.709894 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.709901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.710174 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.711000 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:05:20.711003 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:20.711062 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:05:20.711137 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:20.711673 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:20.711681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:20.711688 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:05:20.711902 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 13:05:20.712192 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":283,"threads_started":1}
I20260619 13:05:20.714460 21689 heartbeater.cc:347] Connected to a master server at 127.1.87.126:35681
I20260619 13:05:20.714727 21689 heartbeater.cc:464] Registering TS with master...
I20260619 13:05:20.715469 21689 heartbeater.cc:511] Master 127.1.87.126:35681 requested a full tablet report, sending...
I20260619 13:05:20.716877 21242 ts_manager.cc:194] Registered new tserver with Master: 931b8017824e400790efba6d422b2687 (127.1.87.67:34771)
I20260619 13:05:20.717770 21242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.87.67:42397
I20260619 13:05:20.723143 1373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:05:20.733209 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.727419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.1.87.126:35681)
0619 13:05:20.727587 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.727591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.728115 (+ 524us) server_negotiation.cc:407] Connection header received
0619 13:05:20.728331 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.728335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.728409 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.728492 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:20.729014 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.729766 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.731156 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.731606 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.731708 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.732677 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:20.732682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:20.732685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:20.732708 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:20.732737 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:20.732748 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:20.732816 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.732930 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.732997 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:05:20.736901 21242 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:05:20.743790 21242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:05:20.771296 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.757274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42819 (local address 127.0.0.1:37084)
0619 13:05:20.758994 (+ 1720us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:20.759017 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:20.759144 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:20.760503 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:20.760511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:20.760539 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:20.761053 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.761069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.763867 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.763877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.767084 (+ 3207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.767108 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.770245 (+ 3137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.770254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.770521 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.770575 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 13:05:20.770735 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 13:05:20.770945 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"thread_start_us":1365,"threads_started":1}
I20260619 13:05:20.772452 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.757948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.1.87.65:42819)
0619 13:05:20.758684 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.758693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.759459 (+ 766us) server_negotiation.cc:407] Connection header received
0619 13:05:20.759691 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.759700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.759951 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.760124 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:20.761291 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.763605 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.767408 (+ 3803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.769483 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.769671 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.770069 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.771885 (+ 1816us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.772088 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":365,"threads_started":1}
I20260619 13:05:20.776747 21363 tablet_service.cc:1467] Processing CreateTablet for tablet 094c7905cf1d4241b910282a7c9ef7a3 (DEFAULT_TABLE table=test-table [id=32e735d94f0b4d6dbbaa9922b205b953]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:20.778560 21363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 094c7905cf1d4241b910282a7c9ef7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:20.778892 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.757133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.1.87.66:39001)
0619 13:05:20.770702 (+ 13569us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.770713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.770747 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:05:20.770854 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.770863 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.771150 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.771409 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:20.772214 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.773975 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.776264 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.777617 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.777770 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.778008 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.778223 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.778535 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13377,"thread_start_us":14451,"threads_started":1}
I20260619 13:05:20.779011 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.756595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.66:39001 (local address 127.0.0.1:47898)
0619 13:05:20.757342 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:20.757395 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:20.757567 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:20.771388 (+ 13821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:20.771393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:20.771416 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:20.771781 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.771792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.774108 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.774116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.776050 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.776071 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.777826 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.777833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.778012 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.778049 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:05:20.778155 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:05:20.778305 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":25,"thread_start_us":357,"threads_started":1}
I20260619 13:05:20.780074 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.758978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:34771 (local address 127.0.0.1:42604)
0619 13:05:20.771043 (+ 12065us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:20.771071 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:20.771232 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:20.773010 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:20.773015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:20.773041 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:20.773491 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.773503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.775375 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.775382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.777249 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.777262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.779260 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.779267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.779488 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.779520 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:05:20.779608 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:05:20.779745 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11877,"mutex_wait_us":24,"thread_start_us":12362,"threads_started":1}
I20260619 13:05:20.780504 21428 heartbeater.cc:503] Master 127.1.87.126:35681 was elected leader, sending a full tablet report...
I20260619 13:05:20.780830 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.759399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.1.87.67:34771)
0619 13:05:20.772278 (+ 12879us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.772287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.772322 (+ 35us) server_negotiation.cc:407] Connection header received
0619 13:05:20.772416 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.772426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.772701 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.773026 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:20.773690 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.775213 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.777449 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.778726 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.778860 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.779127 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.780356 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.780475 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12690,"thread_start_us":12526,"threads_started":1}
I20260619 13:05:20.783500 21494 tablet_service.cc:1467] Processing CreateTablet for tablet 094c7905cf1d4241b910282a7c9ef7a3 (DEFAULT_TABLE table=test-table [id=32e735d94f0b4d6dbbaa9922b205b953]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:20.784502 21624 tablet_service.cc:1467] Processing CreateTablet for tablet 094c7905cf1d4241b910282a7c9ef7a3 (DEFAULT_TABLE table=test-table [id=32e735d94f0b4d6dbbaa9922b205b953]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:05:20.785331 21494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 094c7905cf1d4241b910282a7c9ef7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:20.785785 21624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 094c7905cf1d4241b910282a7c9ef7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:20.792764 21707 tablet_bootstrap.cc:492] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: Bootstrap starting.
I20260619 13:05:20.796938 21707 tablet_bootstrap.cc:654] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:20.798158 21707 log.cc:824] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:20.801172 21707 tablet_bootstrap.cc:492] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: No bootstrap required, opened a new log
I20260619 13:05:20.801225 21710 tablet_bootstrap.cc:492] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: Bootstrap starting.
I20260619 13:05:20.801431 21707 ts_tablet_manager.cc:1397] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 13:05:20.802062 21709 tablet_bootstrap.cc:492] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: Bootstrap starting.
I20260619 13:05:20.805241 21710 tablet_bootstrap.cc:654] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:20.805281 21709 tablet_bootstrap.cc:654] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:20.806205 21710 log.cc:824] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:20.806394 21709 log.cc:824] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:20.809224 21709 tablet_bootstrap.cc:492] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: No bootstrap required, opened a new log
I20260619 13:05:20.809410 21710 tablet_bootstrap.cc:492] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: No bootstrap required, opened a new log
I20260619 13:05:20.809656 21709 ts_tablet_manager.cc:1397] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260619 13:05:20.809724 21710 ts_tablet_manager.cc:1397] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260619 13:05:20.812322 21707 raft_consensus.cc:348] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } }
I20260619 13:05:20.812927 21707 raft_consensus.cc:374] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:20.813063 21707 raft_consensus.cc:729] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e8e8cb54cc4a77a8f9b0161b0158c3, State: Initialized, Role: FOLLOWER
I20260619 13:05:20.813853 21707 consensus_queue.cc:260] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } }
I20260619 13:05:20.818009 21707 ts_tablet_manager.cc:1428] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260619 13:05:20.818318 21709 raft_consensus.cc:348] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } }
I20260619 13:05:20.818770 21709 raft_consensus.cc:374] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:20.818897 21709 raft_consensus.cc:729] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd23e88dc94a479496dbe9559774d085, State: Initialized, Role: FOLLOWER
I20260619 13:05:20.819844 21709 consensus_queue.cc:260] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } }
I20260619 13:05:20.820281 21714 raft_consensus.cc:482] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:05:20.820371 21710 raft_consensus.cc:348] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } }
I20260619 13:05:20.820578 21714 raft_consensus.cc:504] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } }
I20260619 13:05:20.820856 21710 raft_consensus.cc:374] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:20.821000 21710 raft_consensus.cc:729] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931b8017824e400790efba6d422b2687, State: Initialized, Role: FOLLOWER
I20260619 13:05:20.821774 21710 consensus_queue.cc:260] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } }
I20260619 13:05:20.829197 21689 heartbeater.cc:503] Master 127.1.87.126:35681 was elected leader, sending a full tablet report...
I20260619 13:05:20.829594 21714 leader_election.cc:288] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40e8e8cb54cc4a77a8f9b0161b0158c3 (127.1.87.65:42819), 931b8017824e400790efba6d422b2687 (127.1.87.67:34771)
I20260619 13:05:20.830520 21710 ts_tablet_manager.cc:1428] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: Time spent starting tablet: real 0.021s user 0.018s sys 0.001s
I20260619 13:05:20.830706 21559 heartbeater.cc:503] Master 127.1.87.126:35681 was elected leader, sending a full tablet report...
I20260619 13:05:20.838227 21709 ts_tablet_manager.cc:1428] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: Time spent starting tablet: real 0.028s user 0.016s sys 0.012s
I20260619 13:05:20.841894 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.823359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.65:42819 (local address 127.1.87.66:39065)
0619 13:05:20.835012 (+ 11653us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:20.835035 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:20.835258 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:20.835846 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:20.835851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:20.835876 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:20.836650 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.836670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.838052 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.838055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.840013 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.840025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.841194 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.841216 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.841394 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.841436 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 13:05:20.841551 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 13:05:20.841673 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11484,"thread_start_us":11690,"threads_started":1}
I20260619 13:05:20.841955 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.824001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:39065 (local address 127.1.87.65:42819)
0619 13:05:20.824221 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.824230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.835126 (+ 10896us) server_negotiation.cc:407] Connection header received
0619 13:05:20.835462 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.835469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.835574 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.835643 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:20.836845 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.837835 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.840216 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.840982 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.841122 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.841233 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.841578 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.841723 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:05:20.842885 21383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094c7905cf1d4241b910282a7c9ef7a3" candidate_uuid: "fd23e88dc94a479496dbe9559774d085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" is_pre_election: true
I20260619 13:05:20.843221 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.835685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.87.66:50017 (local address 127.1.87.67:34771)
0619 13:05:20.835918 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.835927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.837176 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 13:05:20.837412 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.837417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.837487 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.837547 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:05:20.838499 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.839366 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.841264 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.842353 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.842493 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.842619 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.842870 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.842990 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:05:20.843246 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.835880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.67:34771 (local address 127.1.87.66:50017)
0619 13:05:20.837083 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:20.837105 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:20.837265 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:20.837743 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:20.837747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:20.837771 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:05:20.838297 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.838316 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.839558 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.839563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.841070 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:20.841082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.842528 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.842534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:20.842719 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.842742 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:05:20.842846 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 13:05:20.842965 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":391,"threads_started":1}
I20260619 13:05:20.843497 21383 raft_consensus.cc:2454] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd23e88dc94a479496dbe9559774d085 in term 0.
I20260619 13:05:20.844262 21448 leader_election.cc:302] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e8e8cb54cc4a77a8f9b0161b0158c3, fd23e88dc94a479496dbe9559774d085; no voters:
I20260619 13:05:20.844287 21644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094c7905cf1d4241b910282a7c9ef7a3" candidate_uuid: "fd23e88dc94a479496dbe9559774d085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931b8017824e400790efba6d422b2687" is_pre_election: true
I20260619 13:05:20.844702 21715 raft_consensus.cc:2781] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:05:20.844838 21715 raft_consensus.cc:482] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:05:20.844779 21644 raft_consensus.cc:2454] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd23e88dc94a479496dbe9559774d085 in term 0.
I20260619 13:05:20.844992 21715 raft_consensus.cc:3037] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:20.847368 21715 raft_consensus.cc:504] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } }
I20260619 13:05:20.848008 21715 leader_election.cc:288] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [CANDIDATE]: Term 1 election: Requested vote from peers 40e8e8cb54cc4a77a8f9b0161b0158c3 (127.1.87.65:42819), 931b8017824e400790efba6d422b2687 (127.1.87.67:34771)
I20260619 13:05:20.848433 21383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094c7905cf1d4241b910282a7c9ef7a3" candidate_uuid: "fd23e88dc94a479496dbe9559774d085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3"
I20260619 13:05:20.848510 21644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094c7905cf1d4241b910282a7c9ef7a3" candidate_uuid: "fd23e88dc94a479496dbe9559774d085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931b8017824e400790efba6d422b2687"
I20260619 13:05:20.848660 21383 raft_consensus.cc:3037] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:20.848735 21644 raft_consensus.cc:3037] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:20.850905 21383 raft_consensus.cc:2454] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd23e88dc94a479496dbe9559774d085 in term 1.
I20260619 13:05:20.850970 21644 raft_consensus.cc:2454] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd23e88dc94a479496dbe9559774d085 in term 1.
I20260619 13:05:20.851440 21448 leader_election.cc:302] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 931b8017824e400790efba6d422b2687, fd23e88dc94a479496dbe9559774d085; no voters:
I20260619 13:05:20.851678 21715 raft_consensus.cc:2781] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:20.851861 21715 raft_consensus.cc:686] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 1 LEADER]: Becoming Leader. State: Replica: fd23e88dc94a479496dbe9559774d085, State: Running, Role: LEADER
I20260619 13:05:20.852340 21715 consensus_queue.cc:237] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } }
I20260619 13:05:20.856868 21241 catalog_manager.cc:5495] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd23e88dc94a479496dbe9559774d085 (127.1.87.66). New cstate: current_term: 1 leader_uuid: "fd23e88dc94a479496dbe9559774d085" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd23e88dc94a479496dbe9559774d085" member_type: VOTER last_known_addr { host: "127.1.87.66" port: 39001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 } health_report { overall_health: UNKNOWN } } }
I20260619 13:05:20.884809 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.879419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.1.87.66:39001)
0619 13:05:20.879627 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.879633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.879773 (+ 140us) server_negotiation.cc:407] Connection header received
0619 13:05:20.880014 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.880018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.880105 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.880268 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:05:20.880631 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.881323 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.882674 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.883087 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.883238 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.883746 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:05:20.884250 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:05:20.884416 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.884466 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.884548 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:05:20.892665 21383 raft_consensus.cc:1261] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [term 1 FOLLOWER]: Refusing update from remote peer fd23e88dc94a479496dbe9559774d085: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:20.892663 21644 raft_consensus.cc:1261] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [term 1 FOLLOWER]: Refusing update from remote peer fd23e88dc94a479496dbe9559774d085: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:20.893579 21714 consensus_queue.cc:1035] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8e8cb54cc4a77a8f9b0161b0158c3" member_type: VOTER last_known_addr { host: "127.1.87.65" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:20.893898 21715 consensus_queue.cc:1035] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "931b8017824e400790efba6d422b2687" member_type: VOTER last_known_addr { host: "127.1.87.67" port: 34771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:20.907179 21723 mvcc.cc:205] Tried to move back new op lower bound from 7298557218366795776 to 7298557218216243200. Current Snapshot: MvccSnapshot[applied={T|T < 7298557218366795776}]
I20260619 13:05:20.910013 21722 mvcc.cc:205] Tried to move back new op lower bound from 7298557218366795776 to 7298557218216243200. Current Snapshot: MvccSnapshot[applied={T|T < 7298557218366795776}]
I20260619 13:05:20.918912 21241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:20.919168 21241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:05:20.923003 21241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 973593d19eda4854be2c9abdd83cc094: Sending DeleteTablet for 3 replicas of tablet 094c7905cf1d4241b910282a7c9ef7a3
I20260619 13:05:20.924314 21494 tablet_service.cc:1514] Processing DeleteTablet for tablet 094c7905cf1d4241b910282a7c9ef7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:20 UTC) from {username='slave'} at 127.0.0.1:47898
I20260619 13:05:20.924391 21624 tablet_service.cc:1514] Processing DeleteTablet for tablet 094c7905cf1d4241b910282a7c9ef7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:20 UTC) from {username='slave'} at 127.0.0.1:42604
I20260619 13:05:20.925343 21363 tablet_service.cc:1514] Processing DeleteTablet for tablet 094c7905cf1d4241b910282a7c9ef7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:05:20 UTC) from {username='slave'} at 127.0.0.1:37084
I20260619 13:05:20.925514 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21301
I20260619 13:05:20.937981 21733 tablet_replica.cc:330] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: stopping tablet replica
I20260619 13:05:20.940272 21733 raft_consensus.cc:2229] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:20.941581 21731 tablet_replica.cc:330] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: stopping tablet replica
I20260619 13:05:20.942145 21731 raft_consensus.cc:2229] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:05:20.942906 21731 raft_consensus.cc:2258] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:20.943768 21733 raft_consensus.cc:2258] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:20.944793 21732 tablet_replica.cc:330] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: stopping tablet replica
I20260619 13:05:20.945246 21732 raft_consensus.cc:2229] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:05:20.945655 21731 ts_tablet_manager.cc:1905] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:20.945777 21732 raft_consensus.cc:2258] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:05:20.945873 21733 ts_tablet_manager.cc:1905] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:20.947758 21732 ts_tablet_manager.cc:1905] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:05:20.953364 21733 ts_tablet_manager.cc:1918] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:20.953715 21733 log.cc:1196] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-0/wal/wals/094c7905cf1d4241b910282a7c9ef7a3
I20260619 13:05:20.954736 21733 ts_tablet_manager.cc:1939] T 094c7905cf1d4241b910282a7c9ef7a3 P 40e8e8cb54cc4a77a8f9b0161b0158c3: Deleting consensus metadata
I20260619 13:05:20.955235 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:20.926754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.1.87.65:42819)
0619 13:05:20.926971 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:20.926981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:20.937026 (+ 10045us) server_negotiation.cc:407] Connection header received
0619 13:05:20.938561 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:20.938572 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:20.938709 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:20.938809 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:20.940663 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.942232 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.947819 (+ 5587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:20.948508 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:20.948622 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:20.953236 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:20.953247 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:20.953267 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:20.954190 (+ 923us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:20.954265 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:20.954295 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:20.954414 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:20.954702 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 13:05:20.954845 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:05:20.955823 21732 ts_tablet_manager.cc:1918] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:20.956045 21732 log.cc:1196] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-2/wal/wals/094c7905cf1d4241b910282a7c9ef7a3
I20260619 13:05:20.956524 21423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:20.956622 21732 ts_tablet_manager.cc:1939] T 094c7905cf1d4241b910282a7c9ef7a3 P 931b8017824e400790efba6d422b2687: Deleting consensus metadata
I20260619 13:05:20.957079 21227 catalog_manager.cc:4841] TS 40e8e8cb54cc4a77a8f9b0161b0158c3 (127.1.87.65:42819): tablet 094c7905cf1d4241b910282a7c9ef7a3 (table test-table [id=32e735d94f0b4d6dbbaa9922b205b953]) successfully deleted
I20260619 13:05:20.958338 21227 catalog_manager.cc:4841] TS 931b8017824e400790efba6d422b2687 (127.1.87.67:34771): tablet 094c7905cf1d4241b910282a7c9ef7a3 (table test-table [id=32e735d94f0b4d6dbbaa9922b205b953]) successfully deleted
I20260619 13:05:20.963536 21731 ts_tablet_manager.cc:1918] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:05:20.963739 21731 log.cc:1196] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: Deleting WAL directory at /tmp/dist-test-taskas476m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874004469744-1373-0/minicluster-data/ts-1/wal/wals/094c7905cf1d4241b910282a7c9ef7a3
I20260619 13:05:20.964370 21731 ts_tablet_manager.cc:1939] T 094c7905cf1d4241b910282a7c9ef7a3 P fd23e88dc94a479496dbe9559774d085: Deleting consensus metadata
I20260619 13:05:20.966501 21226 catalog_manager.cc:4841] TS fd23e88dc94a479496dbe9559774d085 (127.1.87.66:39001): tablet 094c7905cf1d4241b910282a7c9ef7a3 (table test-table [id=32e735d94f0b4d6dbbaa9922b205b953]) successfully deleted
I20260619 13:05:21.054127 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 21301
I20260619 13:05:21.072584 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21432
I20260619 13:05:21.081002 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:21.073422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.1.87.66:39001)
0619 13:05:21.073589 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:21.073594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:21.073755 (+ 161us) server_negotiation.cc:407] Connection header received
0619 13:05:21.073922 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:21.073926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:21.074005 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:21.074095 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:21.074578 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:21.075480 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:21.077289 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:21.077722 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:21.077826 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:21.079128 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:21.079137 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:21.079151 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:21.080115 (+ 964us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:21.080168 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:21.080190 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:21.080278 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:21.080484 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 13:05:21.080604 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 13:05:21.081763 21554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:21.195719 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 21432
I20260619 13:05:21.214887 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21562
I20260619 13:05:21.221683 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:21.215774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.1.87.67:34771)
0619 13:05:21.215953 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:21.215959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:21.216031 (+ 72us) server_negotiation.cc:407] Connection header received
0619 13:05:21.216260 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:21.216264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:21.216350 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:21.216465 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:21.216887 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:21.217608 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:21.218838 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:21.219231 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:21.219393 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:21.220402 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:21.220411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:21.220428 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:21.220963 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:21.221007 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:21.221029 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:21.221163 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:21.221286 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 13:05:21.221390 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:05:21.222327 21684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:21.312567 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 21562
I20260619 13:05:21.329644 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21211
I20260619 13:05:21.330678 21272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:21.440762 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 21211
2026-06-19T13:05:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2772 ms)
[----------] 6 tests from EncryptionPolicyTest (21898 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:21 dist-test-slave-tpsm krb5kdc[21741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:21 dist-test-slave-tpsm krb5kdc[21741](info): set up 2 sockets
Jun 19 13:05:21 dist-test-slave-tpsm krb5kdc[21741](info): commencing operation
krb5kdc: starting...
W20260619 13:05:22.556155 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:22 dist-test-slave-tpsm krb5kdc[21741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:22.806875 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:23.146517 21754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:23.146831 21754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:23.146996 21754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:23.154570 21754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:23.154701 21754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:23.154793 21754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:23.154889 21754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:23.154958 21754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:23.165359 21754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:23.166719 21754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:23.168087 21754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:23.180387 21759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:23.180294 21762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:23.180332 21760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:23.181572 21754 server_base.cc:1034] running on GCE node
I20260619 13:05:23.182482 21754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:23.184051 21754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:23.185312 21754 hybrid_clock.cc:648] HybridClock initialized: now 1781874323185272 us; error 55 us; skew 500 ppm
I20260619 13:05:23.185707 21754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:23 dist-test-slave-tpsm krb5kdc[21741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:23.189121 21754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:23.190768 21754 webserver.cc:460] Webserver started at http://127.0.0.1:46197/ using document root <none> and password file <none>
I20260619 13:05:23.191597 21754 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:23.191749 21754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:23.192049 21754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:23.195050 21754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "153d9c5511d2448ea0137e14245d77ce"
format_stamp: "Formatted at 2026-06-19 13:05:23 on dist-test-slave-tpsm"
server_key: "2fd155adf9ba1fc7f7d8a591f7f793a8"
server_key_iv: "33c2d0ed1456a970558933c4fc7c8f1b"
server_key_version: "encryptionkey@0"
I20260619 13:05:23.195930 21754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "153d9c5511d2448ea0137e14245d77ce"
format_stamp: "Formatted at 2026-06-19 13:05:23 on dist-test-slave-tpsm"
server_key: "2fd155adf9ba1fc7f7d8a591f7f793a8"
server_key_iv: "33c2d0ed1456a970558933c4fc7c8f1b"
server_key_version: "encryptionkey@0"
I20260619 13:05:23.201262 21754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:05:23.204756 21769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:23.206167 21754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:23.206369 21754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "153d9c5511d2448ea0137e14245d77ce"
format_stamp: "Formatted at 2026-06-19 13:05:23 on dist-test-slave-tpsm"
server_key: "2fd155adf9ba1fc7f7d8a591f7f793a8"
server_key_iv: "33c2d0ed1456a970558933c4fc7c8f1b"
server_key_version: "encryptionkey@0"
I20260619 13:05:23.206578 21754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:23.218521 21754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:23.222889 21754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:23.223178 21754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:23.242447 21754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44843
I20260619 13:05:23.242491 21820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44843 every 8 connection(s)
I20260619 13:05:23.243872 21754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:23.247488 21821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:23.252698 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 21754
I20260619 13:05:23.252907 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:23.253718 1373 external_mini_cluster.cc:1428] Setting key 05fb7f87d39035edddf28fbbddddb982
I20260619 13:05:23.258793 21821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce: Bootstrap starting.
I20260619 13:05:23.262550 21821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:23.263935 21821 log.cc:824] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:23.266955 21821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce: No bootstrap required, opened a new log
Jun 19 13:05:23 dist-test-slave-tpsm krb5kdc[21741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:05:23.277715 21821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d9c5511d2448ea0137e14245d77ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260619 13:05:23.278213 21821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:23.278331 21821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153d9c5511d2448ea0137e14245d77ce, State: Initialized, Role: FOLLOWER
I20260619 13:05:23.279059 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:23.255700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37893 (local address 127.0.0.1:44843)
0619 13:05:23.256544 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:23.256555 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:23.256595 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:05:23.258227 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:23.258250 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:23.258628 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:23.259011 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:23.260757 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:23.262044 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:23.264078 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:23.265089 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:23.265289 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:23.269526 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:23.269537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:23.269564 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:23.269618 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:23.273887 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:23.274751 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:23.274761 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:23.274769 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:23.275014 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:23.275614 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:23.275620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:23.275623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:23.276412 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:23.276747 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:23.277149 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 13:05:23.277519 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":408,"threads_started":1}
I20260619 13:05:23.279168 21821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d9c5511d2448ea0137e14245d77ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260619 13:05:23.279479 21821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:23.279654 21821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:23.279879 21821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:23.282759 21821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d9c5511d2448ea0137e14245d77ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260619 13:05:23.283392 21821 leader_election.cc:302] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 153d9c5511d2448ea0137e14245d77ce; no voters:
I20260619 13:05:23.284050 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:23.284070 21821 leader_election.cc:288] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:23.284197 21826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:23.285458 21826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [term 1 LEADER]: Becoming Leader. State: Replica: 153d9c5511d2448ea0137e14245d77ce, State: Running, Role: LEADER
I20260619 13:05:23.286048 21826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d9c5511d2448ea0137e14245d77ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260619 13:05:23.286197 21821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:23.289230 21827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "153d9c5511d2448ea0137e14245d77ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d9c5511d2448ea0137e14245d77ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } } }
I20260619 13:05:23.289740 21827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:23.289070 21828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 153d9c5511d2448ea0137e14245d77ce. Latest consensus state: current_term: 1 leader_uuid: "153d9c5511d2448ea0137e14245d77ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d9c5511d2448ea0137e14245d77ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } } }
I20260619 13:05:23.290298 21828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:23.290743 21836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:23.308156 21836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:23.322129 21836 catalog_manager.cc:1347] Generated new cluster ID: df7f04b94e814b1cba3676841bfd0abe
I20260619 13:05:23.322328 21836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:23.323923 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:23.291686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50231 (local address 127.0.0.1:44843)
0619 13:05:23.291923 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:23.291931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:23.299809 (+ 7878us) server_negotiation.cc:407] Connection header received
0619 13:05:23.300440 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:23.300450 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:23.300563 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:23.300654 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:23.303812 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:23.305547 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:23.308246 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:23.309225 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:23.309395 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:23.313814 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:23.313821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:23.313826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:23.313864 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:23.318114 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:23.319633 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:23.319639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:23.319644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:23.319829 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:23.320567 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:23.320575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:23.320580 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:23.321168 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:23.322863 (+ 1695us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:23.323095 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 13:05:23.323389 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:05:23.341573 21836 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:23.342851 21836 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:23.351570 21836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 153d9c5511d2448ea0137e14245d77ce: Generated new TSK 0
I20260619 13:05:23.352404 21836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:23.367800 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21754
I20260619 13:05:23.369807 21816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:23.465039 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 21754
W20260619 13:05:23.482316 21739 connection.cc:535] client connection to 127.0.0.1:44843 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44843): Connection reset by peer (error 104)
2026-06-19T13:05:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2017 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:23 dist-test-slave-tpsm krb5kdc[21853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:23 dist-test-slave-tpsm krb5kdc[21853](info): set up 2 sockets
Jun 19 13:05:23 dist-test-slave-tpsm krb5kdc[21853](info): commencing operation
krb5kdc: starting...
W20260619 13:05:24.591606 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:24 dist-test-slave-tpsm krb5kdc[21853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:24.852090 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:25.175243 21866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:25.175622 21866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:25.175791 21866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:25.183038 21866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:25.183205 21866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:25.183357 21866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:25.183454 21866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:25.183538 21866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:25.193605 21866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:25.194995 21866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:25.196322 21866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:25.208627 21871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:25.208654 21872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:25.209551 21866 server_base.cc:1034] running on GCE node
W20260619 13:05:25.209630 21874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:25.210518 21866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:25.211943 21866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:25.213181 21866 hybrid_clock.cc:648] HybridClock initialized: now 1781874325213109 us; error 59 us; skew 500 ppm
I20260619 13:05:25.213606 21866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:25 dist-test-slave-tpsm krb5kdc[21853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:25.217145 21866 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:25.218645 21866 webserver.cc:460] Webserver started at http://127.0.0.1:45817/ using document root <none> and password file <none>
I20260619 13:05:25.219416 21866 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:25.219533 21866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:25.219808 21866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:25.222826 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "bafc3196c2fa4f458ab41d2c99a7d33d"
format_stamp: "Formatted at 2026-06-19 13:05:25 on dist-test-slave-tpsm"
server_key: "209695eb69c41ae74bf636c94be64896"
server_key_iv: "f099d3dfc1de8663f40128d43fa43023"
server_key_version: "encryptionkey@0"
I20260619 13:05:25.223670 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "bafc3196c2fa4f458ab41d2c99a7d33d"
format_stamp: "Formatted at 2026-06-19 13:05:25 on dist-test-slave-tpsm"
server_key: "209695eb69c41ae74bf636c94be64896"
server_key_iv: "f099d3dfc1de8663f40128d43fa43023"
server_key_version: "encryptionkey@0"
I20260619 13:05:25.228857 21866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:05:25.232259 21881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:25.233647 21866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 13:05:25.233831 21866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "bafc3196c2fa4f458ab41d2c99a7d33d"
format_stamp: "Formatted at 2026-06-19 13:05:25 on dist-test-slave-tpsm"
server_key: "209695eb69c41ae74bf636c94be64896"
server_key_iv: "f099d3dfc1de8663f40128d43fa43023"
server_key_version: "encryptionkey@0"
I20260619 13:05:25.234041 21866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:25.269416 21866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:25.273998 21866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:25.274326 21866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:25.293105 21866 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43269
I20260619 13:05:25.293150 21932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43269 every 8 connection(s)
I20260619 13:05:25.294440 21866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:25.296437 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 21866
I20260619 13:05:25.296607 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:25.297138 1373 external_mini_cluster.cc:1428] Setting key 0abcbfc143ee30cd61dc1ce361cc62bc
I20260619 13:05:25.298745 21933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:25.310245 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:25.298833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40933 (local address 127.0.0.1:43269)
0619 13:05:25.299666 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:25.299678 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:25.299715 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:05:25.301351 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:25.301371 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:25.301789 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:25.302235 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:25.302745 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:25.304124 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:25.305919 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:25.306542 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:25.306673 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:25.308050 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:25.308060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:25.308074 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:25.308607 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:25.308681 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:25.308707 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:25.308826 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:25.309104 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 13:05:25.309323 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":406,"threads_started":1}
I20260619 13:05:25.311415 21933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d: Bootstrap starting.
I20260619 13:05:25.313791 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:25.314591 21933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:25.315819 21933 log.cc:824] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:25.318401 21933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d: No bootstrap required, opened a new log
I20260619 13:05:25.325649 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:25.319024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55495 (local address 127.0.0.1:43269)
0619 13:05:25.319206 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:25.319211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:25.319589 (+ 378us) server_negotiation.cc:407] Connection header received
0619 13:05:25.319733 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:25.319744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:25.319814 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:25.319912 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:25.320352 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:25.321550 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:25.322975 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:25.323724 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:25.323865 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:25.324894 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:25.324900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:25.324904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:25.324924 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:25.324966 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:25.324996 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:25.325117 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:25.325271 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 13:05:25.325326 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:05:25.328173 21933 raft_consensus.cc:348] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bafc3196c2fa4f458ab41d2c99a7d33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43269 } }
I20260619 13:05:25.328529 21933 raft_consensus.cc:374] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:25.328614 21933 raft_consensus.cc:729] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bafc3196c2fa4f458ab41d2c99a7d33d, State: Initialized, Role: FOLLOWER
I20260619 13:05:25.329088 21933 consensus_queue.cc:260] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bafc3196c2fa4f458ab41d2c99a7d33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43269 } }
I20260619 13:05:25.329295 21933 raft_consensus.cc:388] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:25.329383 21933 raft_consensus.cc:482] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:25.329502 21933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:25.331995 21933 raft_consensus.cc:504] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bafc3196c2fa4f458ab41d2c99a7d33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43269 } }
I20260619 13:05:25.332432 21933 leader_election.cc:302] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bafc3196c2fa4f458ab41d2c99a7d33d; no voters:
I20260619 13:05:25.332911 21933 leader_election.cc:288] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:25.333081 21943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:25.333809 21943 raft_consensus.cc:686] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [term 1 LEADER]: Becoming Leader. State: Replica: bafc3196c2fa4f458ab41d2c99a7d33d, State: Running, Role: LEADER
I20260619 13:05:25.334364 21943 consensus_queue.cc:237] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bafc3196c2fa4f458ab41d2c99a7d33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43269 } }
I20260619 13:05:25.334728 21933 sys_catalog.cc:564] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:25.337126 21944 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bafc3196c2fa4f458ab41d2c99a7d33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bafc3196c2fa4f458ab41d2c99a7d33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43269 } } }
I20260619 13:05:25.337239 21945 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bafc3196c2fa4f458ab41d2c99a7d33d. Latest consensus state: current_term: 1 leader_uuid: "bafc3196c2fa4f458ab41d2c99a7d33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bafc3196c2fa4f458ab41d2c99a7d33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43269 } } }
I20260619 13:05:25.337522 21944 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:25.337527 21945 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:25.339968 21950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:25.345189 21950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:25.353132 21950 catalog_manager.cc:1347] Generated new cluster ID: d108b8cac4a54a568bcb62fd6a8e494d
I20260619 13:05:25.353241 21950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:25.364465 21950 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:25.366285 21950 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:25.382524 21950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bafc3196c2fa4f458ab41d2c99a7d33d: Generated new TSK 0
I20260619 13:05:25.383637 21950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:25.402559 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21866
W20260619 13:05:25.403716 21928 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40933
W20260619 13:05:25.404441 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21866: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:25.404613 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 21866
W20260619 13:05:25.421813 21851 connection.cc:535] client connection to 127.0.0.1:43269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43269): Connection reset by peer (error 104)
2026-06-19T13:05:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:25 dist-test-slave-tpsm krb5kdc[21964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:25 dist-test-slave-tpsm krb5kdc[21964](info): set up 2 sockets
Jun 19 13:05:25 dist-test-slave-tpsm krb5kdc[21964](info): commencing operation
krb5kdc: starting...
W20260619 13:05:26.516067 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:26 dist-test-slave-tpsm krb5kdc[21964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:26.770025 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:27.108621 21977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:27.108927 21977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:27.109023 21977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:27.116357 21977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:27.116475 21977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:27.116523 21977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:27.116578 21977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:27.116624 21977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:27.126751 21977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:27.128036 21977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:27.129248 21977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:27.140872 21982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:27.140820 21983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:27.141136 21985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:27.142138 21977 server_base.cc:1034] running on GCE node
I20260619 13:05:27.142971 21977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:27.144500 21977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:27.145764 21977 hybrid_clock.cc:648] HybridClock initialized: now 1781874327145725 us; error 53 us; skew 500 ppm
I20260619 13:05:27.146144 21977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:27 dist-test-slave-tpsm krb5kdc[21964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:27.152700 21977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:27.154917 21977 webserver.cc:460] Webserver started at http://127.0.0.1:36665/ using document root <none> and password file <none>
I20260619 13:05:27.155777 21977 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:27.155938 21977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:27.156261 21977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:27.159570 21977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "e20f35f328f74d89a857c1d4c0263e9c"
format_stamp: "Formatted at 2026-06-19 13:05:27 on dist-test-slave-tpsm"
server_key: "1facbb3e62324863e981111dbf67494e"
server_key_iv: "55f020125d55e162945da4d29049bd99"
server_key_version: "encryptionkey@0"
I20260619 13:05:27.160698 21977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "e20f35f328f74d89a857c1d4c0263e9c"
format_stamp: "Formatted at 2026-06-19 13:05:27 on dist-test-slave-tpsm"
server_key: "1facbb3e62324863e981111dbf67494e"
server_key_iv: "55f020125d55e162945da4d29049bd99"
server_key_version: "encryptionkey@0"
I20260619 13:05:27.166894 21977 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 13:05:27.170509 21992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:27.171908 21977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:27.172099 21977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "e20f35f328f74d89a857c1d4c0263e9c"
format_stamp: "Formatted at 2026-06-19 13:05:27 on dist-test-slave-tpsm"
server_key: "1facbb3e62324863e981111dbf67494e"
server_key_iv: "55f020125d55e162945da4d29049bd99"
server_key_version: "encryptionkey@0"
I20260619 13:05:27.172328 21977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:27.178481 21977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:27.183032 21977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:27.183382 21977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:27.202692 21977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35547
I20260619 13:05:27.202741 22043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35547 every 8 connection(s)
I20260619 13:05:27.204180 21977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:27.206475 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 21977
I20260619 13:05:27.206653 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:27.207191 1373 external_mini_cluster.cc:1428] Setting key 3586911448186249c3ab3b37954d6364
I20260619 13:05:27.208454 22044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:27.215056 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:27.209146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41993 (local address 127.0.0.1:35547)
0619 13:05:27.209940 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:27.209952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:27.209990 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:05:27.211665 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:27.211685 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:27.212024 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:27.212433 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:27.213477 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:27.213483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:27.213498 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:27.213514 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:27.213582 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:27.213604 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:27.213843 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:27.213993 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 13:05:27.214167 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":350,"threads_started":1}
I20260619 13:05:27.218580 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:27.220976 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c: Bootstrap starting.
I20260619 13:05:27.225538 22044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:27.226737 22044 log.cc:824] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:27.228223 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:27.222844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37323 (local address 127.0.0.1:35547)
0619 13:05:27.222996 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:27.223000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:27.226142 (+ 3142us) server_negotiation.cc:407] Connection header received
0619 13:05:27.226340 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:27.226344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:27.226393 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:27.226497 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:27.227520 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:27.227524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:27.227527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:27.227542 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:27.227584 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:27.227602 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:27.227702 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:27.227863 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 13:05:27.227913 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:05:27.229951 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c: No bootstrap required, opened a new log
I20260619 13:05:27.237568 22044 raft_consensus.cc:348] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20f35f328f74d89a857c1d4c0263e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260619 13:05:27.237950 22044 raft_consensus.cc:374] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:27.238051 22044 raft_consensus.cc:729] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20f35f328f74d89a857c1d4c0263e9c, State: Initialized, Role: FOLLOWER
I20260619 13:05:27.238787 22044 consensus_queue.cc:260] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20f35f328f74d89a857c1d4c0263e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260619 13:05:27.239041 22044 raft_consensus.cc:388] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:27.239174 22044 raft_consensus.cc:482] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:27.239382 22044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:27.241498 22044 raft_consensus.cc:504] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20f35f328f74d89a857c1d4c0263e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260619 13:05:27.242009 22044 leader_election.cc:302] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e20f35f328f74d89a857c1d4c0263e9c; no voters:
I20260619 13:05:27.242586 22044 leader_election.cc:288] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:27.242723 22054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:27.243438 22054 raft_consensus.cc:686] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [term 1 LEADER]: Becoming Leader. State: Replica: e20f35f328f74d89a857c1d4c0263e9c, State: Running, Role: LEADER
I20260619 13:05:27.243917 22054 consensus_queue.cc:237] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20f35f328f74d89a857c1d4c0263e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260619 13:05:27.244726 22044 sys_catalog.cc:564] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:27.247161 22055 sys_catalog.cc:455] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e20f35f328f74d89a857c1d4c0263e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20f35f328f74d89a857c1d4c0263e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } } }
I20260619 13:05:27.247576 22055 sys_catalog.cc:458] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:27.247504 22056 sys_catalog.cc:455] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e20f35f328f74d89a857c1d4c0263e9c. Latest consensus state: current_term: 1 leader_uuid: "e20f35f328f74d89a857c1d4c0263e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20f35f328f74d89a857c1d4c0263e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } } }
I20260619 13:05:27.247786 22056 sys_catalog.cc:458] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c [sys.catalog]: This master's current role is: LEADER
W20260619 13:05:27.257602 22070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:05:27.257756 22070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:05:27.258248 22062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:27.259958 22062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:27.268097 22062 catalog_manager.cc:1347] Generated new cluster ID: 469e6bfcb2e34b898c845441e638a41b
I20260619 13:05:27.268196 22062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:27.279376 22062 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:27.279831 22062 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:27.299817 22062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e20f35f328f74d89a857c1d4c0263e9c: Generated new TSK 0
I20260619 13:05:27.300956 22062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:27.307828 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21977
W20260619 13:05:27.309242 22039 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41993
W20260619 13:05:27.310032 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 21977: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:27.310171 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 21977
2026-06-19T13:05:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:27 dist-test-slave-tpsm krb5kdc[22075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:27 dist-test-slave-tpsm krb5kdc[22075](info): set up 2 sockets
Jun 19 13:05:27 dist-test-slave-tpsm krb5kdc[22075](info): commencing operation
krb5kdc: starting...
W20260619 13:05:28.411691 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:28 dist-test-slave-tpsm krb5kdc[22075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:28.670359 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:29.012630 22088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:29.012955 22088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:29.013118 22088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:29.021183 22088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:29.021307 22088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:29.021394 22088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:29.021476 22088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:29.021536 22088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:29.031809 22088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:29.033138 22088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:29.034576 22088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:29.046089 22093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:29.046151 22094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:29.046118 22096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:29.047410 22088 server_base.cc:1034] running on GCE node
I20260619 13:05:29.048141 22088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:29.049582 22088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:29.050817 22088 hybrid_clock.cc:648] HybridClock initialized: now 1781874329050791 us; error 59 us; skew 500 ppm
I20260619 13:05:29.051172 22088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:29 dist-test-slave-tpsm krb5kdc[22075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:29.054293 22088 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:29.055845 22088 webserver.cc:460] Webserver started at http://127.0.0.1:41713/ using document root <none> and password file <none>
I20260619 13:05:29.056521 22088 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:29.056660 22088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:29.056963 22088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:29.060284 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "7626fff5f1c446e1979a4838a63cfc8b"
format_stamp: "Formatted at 2026-06-19 13:05:29 on dist-test-slave-tpsm"
server_key: "b879054d0fa715a7c79f8332e564004e"
server_key_iv: "5113cbd74399b5861dcca768f8b9b388"
server_key_version: "encryptionkey@0"
I20260619 13:05:29.061034 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "7626fff5f1c446e1979a4838a63cfc8b"
format_stamp: "Formatted at 2026-06-19 13:05:29 on dist-test-slave-tpsm"
server_key: "b879054d0fa715a7c79f8332e564004e"
server_key_iv: "5113cbd74399b5861dcca768f8b9b388"
server_key_version: "encryptionkey@0"
I20260619 13:05:29.066143 22088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:05:29.069651 22103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:29.070992 22088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:05:29.071174 22088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "7626fff5f1c446e1979a4838a63cfc8b"
format_stamp: "Formatted at 2026-06-19 13:05:29 on dist-test-slave-tpsm"
server_key: "b879054d0fa715a7c79f8332e564004e"
server_key_iv: "5113cbd74399b5861dcca768f8b9b388"
server_key_version: "encryptionkey@0"
I20260619 13:05:29.071421 22088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:29.096622 22088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:29.101069 22088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:29.101365 22088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:29.120337 22088 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33463
I20260619 13:05:29.120386 22154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33463 every 8 connection(s)
I20260619 13:05:29.121801 22088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:29.125586 22155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:29.125960 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 22088
I20260619 13:05:29.126152 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:29.126706 1373 external_mini_cluster.cc:1428] Setting key 92532f67258d3f8dedb5a918cf4e2a64
I20260619 13:05:29.140370 22155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b: Bootstrap starting.
Jun 19 13:05:29 dist-test-slave-tpsm krb5kdc[22075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:05:29.144107 22155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:29.145350 22155 log.cc:824] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:29.148716 22155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b: No bootstrap required, opened a new log
I20260619 13:05:29.152486 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:29.128558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53721 (local address 127.0.0.1:33463)
0619 13:05:29.129413 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:29.129424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:29.129463 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:05:29.131938 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:29.131962 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:29.132376 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:29.132728 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:29.134745 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:29.136113 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:29.137989 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:29.138691 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:29.138834 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:29.143800 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:29.143807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:29.143822 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:29.143858 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:29.147715 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:29.148502 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:29.148510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:29.148517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:29.148737 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:29.149218 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:29.149225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:29.149230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:29.149906 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:29.150182 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:29.150645 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 13:05:29.151099 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":387,"threads_started":1}
I20260619 13:05:29.156209 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:29.158612 22155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7626fff5f1c446e1979a4838a63cfc8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } }
I20260619 13:05:29.159111 22155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:29.159221 22155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7626fff5f1c446e1979a4838a63cfc8b, State: Initialized, Role: FOLLOWER
I20260619 13:05:29.160090 22155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7626fff5f1c446e1979a4838a63cfc8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } }
I20260619 13:05:29.160429 22155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:29.160589 22155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:29.160800 22155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:29.163678 22155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7626fff5f1c446e1979a4838a63cfc8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } }
I20260619 13:05:29.164213 22155 leader_election.cc:302] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7626fff5f1c446e1979a4838a63cfc8b; no voters:
I20260619 13:05:29.164855 22155 leader_election.cc:288] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:29.164954 22165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:29.167624 22165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [term 1 LEADER]: Becoming Leader. State: Replica: 7626fff5f1c446e1979a4838a63cfc8b, State: Running, Role: LEADER
I20260619 13:05:29.167656 22155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:29.168324 22165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7626fff5f1c446e1979a4838a63cfc8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } }
I20260619 13:05:29.176687 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:29.162157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42081 (local address 127.0.0.1:33463)
0619 13:05:29.162357 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:29.162364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:29.162887 (+ 523us) server_negotiation.cc:407] Connection header received
0619 13:05:29.163018 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:29.163023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:29.163098 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:29.163218 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:29.164894 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:29.166052 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:29.167361 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:29.168070 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:29.168209 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:29.170805 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:29.170811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:29.170822 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:29.170853 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:29.173833 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:29.174626 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:29.174633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:29.174636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:29.174761 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:29.175120 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:29.175126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:29.175129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:29.175570 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:29.175844 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:29.176255 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 13:05:29.176429 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:05:29.177767 22166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7626fff5f1c446e1979a4838a63cfc8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7626fff5f1c446e1979a4838a63cfc8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } } }
I20260619 13:05:29.178210 22166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:29.178795 22167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7626fff5f1c446e1979a4838a63cfc8b. Latest consensus state: current_term: 1 leader_uuid: "7626fff5f1c446e1979a4838a63cfc8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7626fff5f1c446e1979a4838a63cfc8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33463 } } }
I20260619 13:05:29.179091 22167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:29.184804 22173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:29.190198 22173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:29.199558 22173 catalog_manager.cc:1347] Generated new cluster ID: 10e5f74d68164cac8b215d4bb3d5ae37
I20260619 13:05:29.199690 22173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:29.215278 22173 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:29.217139 22173 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:29.226310 22173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7626fff5f1c446e1979a4838a63cfc8b: Generated new TSK 0
I20260619 13:05:29.227617 22173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:29.238286 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22088
I20260619 13:05:29.240314 22150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:29.331277 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 22088
2026-06-19T13:05:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:29 dist-test-slave-tpsm krb5kdc[22187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:29 dist-test-slave-tpsm krb5kdc[22187](info): set up 2 sockets
Jun 19 13:05:29 dist-test-slave-tpsm krb5kdc[22187](info): commencing operation
krb5kdc: starting...
W20260619 13:05:30.443763 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:30 dist-test-slave-tpsm krb5kdc[22187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:30.702275 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:31.028275 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:31.028575 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:31.028744 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:31.036360 22200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:31.036476 22200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:31.036572 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:31.036670 22200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:31.036746 22200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:31.046635 22200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:31.047976 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:31.049295 22200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:31.061162 22206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:31.061239 22208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:31.061197 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:31.061969 22200 server_base.cc:1034] running on GCE node
I20260619 13:05:31.062779 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:31.064229 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:31.065433 22200 hybrid_clock.cc:648] HybridClock initialized: now 1781874331065382 us; error 37 us; skew 500 ppm
I20260619 13:05:31.065778 22200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:31 dist-test-slave-tpsm krb5kdc[22187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:31.069427 22200 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:31.070947 22200 webserver.cc:460] Webserver started at http://127.0.0.1:33719/ using document root <none> and password file <none>
I20260619 13:05:31.071841 22200 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:31.071998 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:31.072338 22200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:31.075306 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8"
format_stamp: "Formatted at 2026-06-19 13:05:31 on dist-test-slave-tpsm"
server_key: "d062c48cc82dcb330c5699443d00c4c7"
server_key_iv: "846fa1b42f16b6600418849ea1f47100"
server_key_version: "encryptionkey@0"
I20260619 13:05:31.076195 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8"
format_stamp: "Formatted at 2026-06-19 13:05:31 on dist-test-slave-tpsm"
server_key: "d062c48cc82dcb330c5699443d00c4c7"
server_key_iv: "846fa1b42f16b6600418849ea1f47100"
server_key_version: "encryptionkey@0"
I20260619 13:05:31.081352 22200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 13:05:31.085176 22215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:31.086560 22200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:05:31.086774 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8"
format_stamp: "Formatted at 2026-06-19 13:05:31 on dist-test-slave-tpsm"
server_key: "d062c48cc82dcb330c5699443d00c4c7"
server_key_iv: "846fa1b42f16b6600418849ea1f47100"
server_key_version: "encryptionkey@0"
I20260619 13:05:31.086957 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:31.124424 22200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:31.128564 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:31.128876 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:31.147785 22200 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42709
I20260619 13:05:31.147820 22266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42709 every 8 connection(s)
I20260619 13:05:31.149106 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:31.152685 22267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:31.156082 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 22200
I20260619 13:05:31.156270 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:31.156803 1373 external_mini_cluster.cc:1428] Setting key fa48eea6e207e119267cb36e172aeeed
I20260619 13:05:31.164986 22267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8: Bootstrap starting.
I20260619 13:05:31.168471 22267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:31.169669 22267 log.cc:824] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:31.170343 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:31.158628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57087 (local address 127.0.0.1:42709)
0619 13:05:31.159479 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:31.159491 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:31.159541 (+ 50us) server_negotiation.cc:407] Connection header received
0619 13:05:31.161267 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:31.161288 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:31.161704 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:31.162027 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:31.162590 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:31.164025 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:31.165806 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:31.166577 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:31.166710 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:31.167981 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:31.167988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:31.168000 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:31.168491 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:31.168631 (+ 140us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:31.168666 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:31.168751 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:31.168996 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 13:05:31.169246 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":393,"threads_started":1}
I20260619 13:05:31.172446 22267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8: No bootstrap required, opened a new log
I20260619 13:05:31.173990 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:31.182447 22267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260619 13:05:31.182829 22267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:31.182925 22267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fdfc1e255734cc0ab405d6aa1ca1da8, State: Initialized, Role: FOLLOWER
I20260619 13:05:31.183573 22267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260619 13:05:31.183817 22267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:31.183979 22267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:31.184132 22267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:31.186026 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:31.179054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59085 (local address 127.0.0.1:42709)
0619 13:05:31.179246 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:31.179251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:31.179649 (+ 398us) server_negotiation.cc:407] Connection header received
0619 13:05:31.179812 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:31.179818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:31.179889 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:31.179982 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:31.180417 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:31.181535 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:31.183138 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:31.183717 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:31.183871 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:31.185305 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:31.185311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:31.185317 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:31.185336 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:31.185373 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:31.185389 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:31.185511 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:31.185655 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 13:05:31.185741 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:05:31.186672 22267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260619 13:05:31.187155 22267 leader_election.cc:302] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fdfc1e255734cc0ab405d6aa1ca1da8; no voters:
I20260619 13:05:31.187827 22267 leader_election.cc:288] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:31.187966 22277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:31.188654 22277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [term 1 LEADER]: Becoming Leader. State: Replica: 2fdfc1e255734cc0ab405d6aa1ca1da8, State: Running, Role: LEADER
I20260619 13:05:31.189247 22277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260619 13:05:31.189821 22267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:31.192056 22279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fdfc1e255734cc0ab405d6aa1ca1da8. Latest consensus state: current_term: 1 leader_uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } } }
I20260619 13:05:31.192492 22279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:31.193277 22278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdfc1e255734cc0ab405d6aa1ca1da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } } }
I20260619 13:05:31.193701 22278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:31.196203 22282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:31.201448 22282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:31.210505 22282 catalog_manager.cc:1347] Generated new cluster ID: c877e37183654e15b3039c4a200833ec
I20260619 13:05:31.210642 22282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:31.218618 22282 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:31.219919 22282 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:31.232082 22282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fdfc1e255734cc0ab405d6aa1ca1da8: Generated new TSK 0
I20260619 13:05:31.233199 22282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:31.249603 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22200
W20260619 13:05:31.250703 22262 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57087
W20260619 13:05:31.251474 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22200: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:31.251633 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 22200
2026-06-19T13:05:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:31 dist-test-slave-tpsm krb5kdc[22298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:31 dist-test-slave-tpsm krb5kdc[22298](info): set up 2 sockets
Jun 19 13:05:31 dist-test-slave-tpsm krb5kdc[22298](info): commencing operation
krb5kdc: starting...
W20260619 13:05:32.355758 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:32 dist-test-slave-tpsm krb5kdc[22298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:32.623970 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:32.964512 22311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:32.964898 22311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:32.965083 22311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:32.972721 22311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:32.972900 22311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:32.973026 22311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:32.973148 22311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:32.973233 22311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:32.983556 22311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:32.984917 22311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:32.986248 22311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:32.998549 22317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:32.998623 22319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:32.998708 22316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:32.999821 22311 server_base.cc:1034] running on GCE node
I20260619 13:05:33.000679 22311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:33.002115 22311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:33.003440 22311 hybrid_clock.cc:648] HybridClock initialized: now 1781874333003388 us; error 33 us; skew 500 ppm
I20260619 13:05:33.003783 22311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:33 dist-test-slave-tpsm krb5kdc[22298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:33.011710 22311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:33.013317 22311 webserver.cc:460] Webserver started at http://127.0.0.1:42117/ using document root <none> and password file <none>
I20260619 13:05:33.014063 22311 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:33.014178 22311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:33.014533 22311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:33.017678 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "5befb7b35356425393aaaa3431c54de4"
format_stamp: "Formatted at 2026-06-19 13:05:33 on dist-test-slave-tpsm"
server_key: "388fff79d2fd7e2d1ae42ce7b59e3d7f"
server_key_iv: "c67874a427affeb569e4d9d84d705a82"
server_key_version: "encryptionkey@0"
I20260619 13:05:33.018468 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "5befb7b35356425393aaaa3431c54de4"
format_stamp: "Formatted at 2026-06-19 13:05:33 on dist-test-slave-tpsm"
server_key: "388fff79d2fd7e2d1ae42ce7b59e3d7f"
server_key_iv: "c67874a427affeb569e4d9d84d705a82"
server_key_version: "encryptionkey@0"
I20260619 13:05:33.023831 22311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:05:33.027209 22326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:33.028517 22311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:33.028707 22311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "5befb7b35356425393aaaa3431c54de4"
format_stamp: "Formatted at 2026-06-19 13:05:33 on dist-test-slave-tpsm"
server_key: "388fff79d2fd7e2d1ae42ce7b59e3d7f"
server_key_iv: "c67874a427affeb569e4d9d84d705a82"
server_key_version: "encryptionkey@0"
I20260619 13:05:33.028914 22311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:33.034662 22311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:33.038719 22311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:33.039031 22311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:33.058423 22311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41581
I20260619 13:05:33.058485 22377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41581 every 8 connection(s)
I20260619 13:05:33.059924 22311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:33.063604 22378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:33.069875 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 22311
I20260619 13:05:33.070081 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:33.070663 1373 external_mini_cluster.cc:1428] Setting key 12a5d553f8d7540730ce06cd9fb41755
I20260619 13:05:33.075965 22378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4: Bootstrap starting.
I20260619 13:05:33.079062 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:33.072982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40707 (local address 127.0.0.1:41581)
0619 13:05:33.073809 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:33.073821 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:33.073862 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:05:33.075721 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:33.075744 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:33.076148 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:33.076496 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:33.077541 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:33.077547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:33.077558 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:33.077573 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:33.077633 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:33.077655 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:33.077830 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:33.077932 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 13:05:33.078138 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":385,"threads_started":1}
I20260619 13:05:33.080546 22378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:33.081804 22378 log.cc:824] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:33.082805 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:33.085228 22378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4: No bootstrap required, opened a new log
I20260619 13:05:33.090567 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:33.087970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36911 (local address 127.0.0.1:41581)
0619 13:05:33.088179 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:33.088186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:33.088620 (+ 434us) server_negotiation.cc:407] Connection header received
0619 13:05:33.088771 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:33.088777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:33.088847 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:33.088965 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:33.089943 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:33.089947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:33.089950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:33.089965 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:33.090005 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:33.090022 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:33.090163 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:33.090254 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 13:05:33.090314 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 13:05:33.095161 22378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5befb7b35356425393aaaa3431c54de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } }
I20260619 13:05:33.095672 22378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:33.095844 22378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5befb7b35356425393aaaa3431c54de4, State: Initialized, Role: FOLLOWER
I20260619 13:05:33.096606 22378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5befb7b35356425393aaaa3431c54de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } }
I20260619 13:05:33.096848 22378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:33.096958 22378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:33.097106 22378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:33.099501 22378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5befb7b35356425393aaaa3431c54de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } }
I20260619 13:05:33.099994 22378 leader_election.cc:302] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5befb7b35356425393aaaa3431c54de4; no voters:
I20260619 13:05:33.100538 22378 leader_election.cc:288] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:33.100697 22388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:33.101423 22388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [term 1 LEADER]: Becoming Leader. State: Replica: 5befb7b35356425393aaaa3431c54de4, State: Running, Role: LEADER
I20260619 13:05:33.101974 22388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5befb7b35356425393aaaa3431c54de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } }
I20260619 13:05:33.102859 22378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:33.108996 22389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5befb7b35356425393aaaa3431c54de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5befb7b35356425393aaaa3431c54de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } } }
I20260619 13:05:33.109440 22389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:33.110379 22399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:33.110154 22390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5befb7b35356425393aaaa3431c54de4. Latest consensus state: current_term: 1 leader_uuid: "5befb7b35356425393aaaa3431c54de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5befb7b35356425393aaaa3431c54de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41581 } } }
I20260619 13:05:33.110630 22390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:33.116139 22399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:33.125057 22399 catalog_manager.cc:1347] Generated new cluster ID: 76f823ec55134c48a2bff314f465fc83
I20260619 13:05:33.125241 22399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:33.166230 22399 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:33.166981 22399 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:33.181033 22399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5befb7b35356425393aaaa3431c54de4: Generated new TSK 0
I20260619 13:05:33.182021 22399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:33.189397 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22311
W20260619 13:05:33.190809 22373 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40707
W20260619 13:05:33.191612 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22311: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:33.191761 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 22311
2026-06-19T13:05:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:33 dist-test-slave-tpsm krb5kdc[22409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:33 dist-test-slave-tpsm krb5kdc[22409](info): set up 2 sockets
Jun 19 13:05:33 dist-test-slave-tpsm krb5kdc[22409](info): commencing operation
krb5kdc: starting...
W20260619 13:05:34.311930 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:34 dist-test-slave-tpsm krb5kdc[22409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:34.558158 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:34.886637 22422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:34.886926 22422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:34.887105 22422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:34.894579 22422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:34.894699 22422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:34.894806 22422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:34.894915 22422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:34.894965 22422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:34.905234 22422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:34.906566 22422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:34.907909 22422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:34.920126 22427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:34.920076 22430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:34.920076 22428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:34.920821 22422 server_base.cc:1034] running on GCE node
I20260619 13:05:34.921649 22422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:34.923143 22422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:34.924350 22422 hybrid_clock.cc:648] HybridClock initialized: now 1781874334924301 us; error 50 us; skew 500 ppm
I20260619 13:05:34.924714 22422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:34 dist-test-slave-tpsm krb5kdc[22409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:34.927970 22422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:34.929507 22422 webserver.cc:460] Webserver started at http://127.0.0.1:45105/ using document root <none> and password file <none>
I20260619 13:05:34.930234 22422 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:34.930382 22422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:34.930691 22422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:34.933776 22422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "b16383d91cbe4ed096e570a551a44768"
format_stamp: "Formatted at 2026-06-19 13:05:34 on dist-test-slave-tpsm"
server_key: "b46fd45a47aa07bd80c23a304b444356"
server_key_iv: "5b4c8719ce7da92a2c21e7aef5fe51c3"
server_key_version: "encryptionkey@0"
I20260619 13:05:34.934556 22422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "b16383d91cbe4ed096e570a551a44768"
format_stamp: "Formatted at 2026-06-19 13:05:34 on dist-test-slave-tpsm"
server_key: "b46fd45a47aa07bd80c23a304b444356"
server_key_iv: "5b4c8719ce7da92a2c21e7aef5fe51c3"
server_key_version: "encryptionkey@0"
I20260619 13:05:34.939939 22422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 13:05:34.943588 22437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:34.944859 22422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:34.945040 22422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "b16383d91cbe4ed096e570a551a44768"
format_stamp: "Formatted at 2026-06-19 13:05:34 on dist-test-slave-tpsm"
server_key: "b46fd45a47aa07bd80c23a304b444356"
server_key_iv: "5b4c8719ce7da92a2c21e7aef5fe51c3"
server_key_version: "encryptionkey@0"
I20260619 13:05:34.945254 22422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:34.974546 22422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:34.978852 22422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:34.979159 22422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:34.998133 22422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34619
I20260619 13:05:34.998181 22488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34619 every 8 connection(s)
I20260619 13:05:34.999522 22422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:35.002569 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 22422
I20260619 13:05:35.002753 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:35.003294 1373 external_mini_cluster.cc:1428] Setting key 9e45fe706d802d97aae8101a616e697c
I20260619 13:05:35.003420 22489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:35.016175 22489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768: Bootstrap starting.
Jun 19 13:05:35 dist-test-slave-tpsm krb5kdc[22409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:05:35.019944 22489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:35.021067 22489 log.cc:824] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:35.024322 22489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768: No bootstrap required, opened a new log
I20260619 13:05:35.026006 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:35.005117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:33511 (local address 127.0.0.1:34619)
0619 13:05:35.005842 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:35.005854 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:35.005891 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:05:35.008077 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:35.008097 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:35.008469 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:35.008836 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:35.010261 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:35.011592 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:35.012820 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:35.013643 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:35.013886 (+ 243us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:35.018429 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:35.018437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:35.018453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:35.018492 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:35.021730 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:35.022626 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:35.022632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:35.022638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:35.022797 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:35.023296 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:35.023299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:35.023301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:35.023847 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:35.024069 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:35.024697 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 13:05:35.025030 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":330,"threads_started":1}
I20260619 13:05:35.029853 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:35.032425 22489 raft_consensus.cc:348] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16383d91cbe4ed096e570a551a44768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } }
I20260619 13:05:35.032945 22489 raft_consensus.cc:374] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:35.033067 22489 raft_consensus.cc:729] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16383d91cbe4ed096e570a551a44768, State: Initialized, Role: FOLLOWER
I20260619 13:05:35.033697 22489 consensus_queue.cc:260] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16383d91cbe4ed096e570a551a44768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } }
I20260619 13:05:35.033941 22489 raft_consensus.cc:388] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:35.034087 22489 raft_consensus.cc:482] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:35.034281 22489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:35.036741 22489 raft_consensus.cc:504] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16383d91cbe4ed096e570a551a44768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } }
I20260619 13:05:35.037249 22489 leader_election.cc:302] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b16383d91cbe4ed096e570a551a44768; no voters:
I20260619 13:05:35.037896 22489 leader_election.cc:288] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:35.038008 22499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:35.039611 22499 raft_consensus.cc:686] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [term 1 LEADER]: Becoming Leader. State: Replica: b16383d91cbe4ed096e570a551a44768, State: Running, Role: LEADER
I20260619 13:05:35.040344 22489 sys_catalog.cc:564] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:35.040271 22499 consensus_queue.cc:237] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16383d91cbe4ed096e570a551a44768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } }
I20260619 13:05:35.049503 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:35.034538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:45505 (local address 127.0.0.1:34619)
0619 13:05:35.034729 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:35.034737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:35.035035 (+ 298us) server_negotiation.cc:407] Connection header received
0619 13:05:35.035241 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:35.035247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:35.035374 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:35.035493 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:35.036870 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:35.037731 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:35.039547 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:35.040190 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:35.040280 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:35.043075 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:35.043080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:35.043083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:35.043111 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:35.046042 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:35.046944 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:35.046951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:35.046959 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:35.047069 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:35.047667 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:35.047671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:35.047675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:35.048137 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:35.048408 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:35.049085 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 13:05:35.049215 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:05:35.051994 22501 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b16383d91cbe4ed096e570a551a44768. Latest consensus state: current_term: 1 leader_uuid: "b16383d91cbe4ed096e570a551a44768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16383d91cbe4ed096e570a551a44768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } } }
I20260619 13:05:35.052362 22501 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:35.052210 22500 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b16383d91cbe4ed096e570a551a44768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16383d91cbe4ed096e570a551a44768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } } }
I20260619 13:05:35.052608 22500 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:35.061167 22509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:35.066496 22509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:35.075433 22509 catalog_manager.cc:1347] Generated new cluster ID: 650695b29c8643988ce051521e9dfae6
I20260619 13:05:35.075567 22509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:35.096369 22509 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:35.098404 22509 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:35.115707 22509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b16383d91cbe4ed096e570a551a44768: Generated new TSK 0
I20260619 13:05:35.116783 22509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:35.122021 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22422
I20260619 13:05:35.123139 22484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:35.214794 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 22422
W20260619 13:05:35.232168 22407 connection.cc:535] client connection to 127.0.0.1:34619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34619): Connection reset by peer (error 104)
2026-06-19T13:05:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:35 dist-test-slave-tpsm krb5kdc[22522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:35 dist-test-slave-tpsm krb5kdc[22522](info): set up 2 sockets
Jun 19 13:05:35 dist-test-slave-tpsm krb5kdc[22522](info): commencing operation
krb5kdc: starting...
W20260619 13:05:36.315939 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:36 dist-test-slave-tpsm krb5kdc[22522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:36.572412 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:36.895306 22535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:36.895684 22535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:36.895870 22535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:36.903128 22535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:36.903285 22535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:36.903419 22535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:36.903532 22535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:36.903601 22535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:36.913506 22535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:36.914842 22535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:36.916152 22535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:36.927585 22541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:36.927801 22543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:36.927654 22540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:36.928579 22535 server_base.cc:1034] running on GCE node
I20260619 13:05:36.929358 22535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:36.930783 22535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:36.932004 22535 hybrid_clock.cc:648] HybridClock initialized: now 1781874336931976 us; error 55 us; skew 500 ppm
I20260619 13:05:36.932339 22535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:36 dist-test-slave-tpsm krb5kdc[22522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:36.935670 22535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:36.937212 22535 webserver.cc:460] Webserver started at http://127.0.0.1:34131/ using document root <none> and password file <none>
I20260619 13:05:36.937863 22535 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:36.937965 22535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:36.938210 22535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:36.941196 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "1003f4e4f77b44bc814585ccfe84c820"
format_stamp: "Formatted at 2026-06-19 13:05:36 on dist-test-slave-tpsm"
server_key: "32360c90a6ea2a4d1729f9eeb9be9371"
server_key_iv: "05157e5cdb72e50f8015075b0fb93ec9"
server_key_version: "encryptionkey@0"
I20260619 13:05:36.942045 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "1003f4e4f77b44bc814585ccfe84c820"
format_stamp: "Formatted at 2026-06-19 13:05:36 on dist-test-slave-tpsm"
server_key: "32360c90a6ea2a4d1729f9eeb9be9371"
server_key_iv: "05157e5cdb72e50f8015075b0fb93ec9"
server_key_version: "encryptionkey@0"
I20260619 13:05:36.947014 22535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:05:36.950441 22550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:36.951759 22535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 13:05:36.951946 22535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "1003f4e4f77b44bc814585ccfe84c820"
format_stamp: "Formatted at 2026-06-19 13:05:36 on dist-test-slave-tpsm"
server_key: "32360c90a6ea2a4d1729f9eeb9be9371"
server_key_iv: "05157e5cdb72e50f8015075b0fb93ec9"
server_key_version: "encryptionkey@0"
I20260619 13:05:36.952124 22535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:36.964605 22535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:36.968362 22535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:36.968658 22535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:36.987056 22535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45799
I20260619 13:05:36.987103 22601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45799 every 8 connection(s)
I20260619 13:05:36.988482 22535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:36.992103 22602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:36.995218 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 22535
I20260619 13:05:36.995435 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:36.995954 1373 external_mini_cluster.cc:1428] Setting key 181c26ba8cc000673d03d3c49394b95b
I20260619 13:05:37.004468 22602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820: Bootstrap starting.
I20260619 13:05:37.008100 22602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:37.009040 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:36.997801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:47137 (local address 127.0.0.1:45799)
0619 13:05:36.998533 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:36.998544 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:36.998590 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:05:37.000370 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:37.000391 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:37.000778 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:37.001081 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:37.001654 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:37.002903 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:37.004761 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:37.005394 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:37.005520 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:37.006818 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:37.006825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:37.006837 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:37.007375 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:37.007454 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:37.007477 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:37.007599 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:37.007865 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 13:05:37.008056 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":358,"threads_started":1}
I20260619 13:05:37.009124 22602 log.cc:824] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:37.012001 22602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820: No bootstrap required, opened a new log
I20260619 13:05:37.012980 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:37.021410 22602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1003f4e4f77b44bc814585ccfe84c820" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260619 13:05:37.021867 22602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:37.021970 22602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1003f4e4f77b44bc814585ccfe84c820, State: Initialized, Role: FOLLOWER
I20260619 13:05:37.022650 22602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1003f4e4f77b44bc814585ccfe84c820" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260619 13:05:37.022913 22602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:37.023078 22602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:37.023255 22602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:37.024461 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:37.017387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:51675 (local address 127.0.0.1:45799)
0619 13:05:37.017674 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:37.017681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:37.018035 (+ 354us) server_negotiation.cc:407] Connection header received
0619 13:05:37.018175 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:37.018180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:37.018253 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:37.018481 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:37.018835 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:37.020363 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:37.021711 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:37.022363 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:37.022495 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:37.023669 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:37.023674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:37.023678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:37.023697 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:37.023759 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:37.023779 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:37.023907 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:37.024051 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 13:05:37.024098 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260619 13:05:37.026091 22602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1003f4e4f77b44bc814585ccfe84c820" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260619 13:05:37.026615 22602 leader_election.cc:302] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1003f4e4f77b44bc814585ccfe84c820; no voters:
I20260619 13:05:37.027156 22602 leader_election.cc:288] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:37.027347 22612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:37.028028 22612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [term 1 LEADER]: Becoming Leader. State: Replica: 1003f4e4f77b44bc814585ccfe84c820, State: Running, Role: LEADER
I20260619 13:05:37.028534 22612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1003f4e4f77b44bc814585ccfe84c820" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260619 13:05:37.029228 22602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:37.031595 22613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1003f4e4f77b44bc814585ccfe84c820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1003f4e4f77b44bc814585ccfe84c820" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } } }
I20260619 13:05:37.032049 22613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:37.033061 22614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1003f4e4f77b44bc814585ccfe84c820. Latest consensus state: current_term: 1 leader_uuid: "1003f4e4f77b44bc814585ccfe84c820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1003f4e4f77b44bc814585ccfe84c820" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } } }
I20260619 13:05:37.033320 22614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:37.033823 22618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:37.038823 22618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:37.048779 22618 catalog_manager.cc:1347] Generated new cluster ID: 9c6304ca39624f69a759e2fd37bb25fd
I20260619 13:05:37.048902 22618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:37.078704 22618 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:37.080255 22618 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:37.091645 22618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1003f4e4f77b44bc814585ccfe84c820: Generated new TSK 0
I20260619 13:05:37.092651 22618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:37.110168 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22535
W20260619 13:05:37.111675 22597 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.191:47137
W20260619 13:05:37.112528 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22535: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:37.112681 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 22535
W20260619 13:05:37.129431 22520 connection.cc:535] client connection to 127.0.0.1:45799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45799): Connection reset by peer (error 104)
2026-06-19T13:05:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:37 dist-test-slave-tpsm krb5kdc[22633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:37 dist-test-slave-tpsm krb5kdc[22633](info): set up 2 sockets
Jun 19 13:05:37 dist-test-slave-tpsm krb5kdc[22633](info): commencing operation
krb5kdc: starting...
W20260619 13:05:38.203817 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:38 dist-test-slave-tpsm krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:38.450131 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:38.779224 22646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:38.779523 22646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:38.779698 22646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:38.786818 22646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:38.786913 22646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:38.787045 22646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:38.787149 22646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:38.787217 22646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:38.796927 22646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:38.798233 22646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:38.799702 22646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:38.811214 22651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:38.811303 22654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:38.811264 22652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:38.812458 22646 server_base.cc:1034] running on GCE node
I20260619 13:05:38.813243 22646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:38.814528 22646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:38.815768 22646 hybrid_clock.cc:648] HybridClock initialized: now 1781874338815720 us; error 38 us; skew 500 ppm
I20260619 13:05:38.816147 22646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:38 dist-test-slave-tpsm krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:38.822589 22646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:38.824231 22646 webserver.cc:460] Webserver started at http://127.0.0.1:35193/ using document root <none> and password file <none>
I20260619 13:05:38.824921 22646 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:38.825054 22646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:38.825371 22646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:38.828321 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "452b2b5aacd54e9da17d9f319504ae05"
format_stamp: "Formatted at 2026-06-19 13:05:38 on dist-test-slave-tpsm"
server_key: "640b791956d1ada2bb0826c72cd50b84"
server_key_iv: "e223cd62da076815ca998395c0bab533"
server_key_version: "encryptionkey@0"
I20260619 13:05:38.829074 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "452b2b5aacd54e9da17d9f319504ae05"
format_stamp: "Formatted at 2026-06-19 13:05:38 on dist-test-slave-tpsm"
server_key: "640b791956d1ada2bb0826c72cd50b84"
server_key_iv: "e223cd62da076815ca998395c0bab533"
server_key_version: "encryptionkey@0"
I20260619 13:05:38.834143 22646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:05:38.837533 22661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:38.838780 22646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 13:05:38.838963 22646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "452b2b5aacd54e9da17d9f319504ae05"
format_stamp: "Formatted at 2026-06-19 13:05:38 on dist-test-slave-tpsm"
server_key: "640b791956d1ada2bb0826c72cd50b84"
server_key_iv: "e223cd62da076815ca998395c0bab533"
server_key_version: "encryptionkey@0"
I20260619 13:05:38.839159 22646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:38.844692 22646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:38.848929 22646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:38.849226 22646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:38.867713 22646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35647
I20260619 13:05:38.867758 22712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35647 every 8 connection(s)
I20260619 13:05:38.869010 22646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:38.872519 22713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:38.875217 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 22646
I20260619 13:05:38.875420 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:38.875857 1373 external_mini_cluster.cc:1428] Setting key 4e2153337cfb878891220ced06ff21ae
I20260619 13:05:38.884377 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:38.877684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:58075 (local address 127.0.0.1:35647)
0619 13:05:38.878429 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:38.878444 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:38.878489 (+ 45us) server_negotiation.cc:407] Connection header received
0619 13:05:38.880764 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:38.880794 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:38.881220 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:38.881625 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:38.882569 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:38.882577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:38.882592 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:38.882614 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:38.882683 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:38.882707 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:38.882916 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:38.883042 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 13:05:38.883239 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":299,"threads_started":1}
I20260619 13:05:38.885309 22713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05: Bootstrap starting.
I20260619 13:05:38.887859 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:38.888551 22713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:38.889516 22713 log.cc:824] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:38.892524 22713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05: No bootstrap required, opened a new log
I20260619 13:05:38.894618 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:38.892198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:41641 (local address 127.0.0.1:35647)
0619 13:05:38.892362 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:38.892367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:38.892739 (+ 372us) server_negotiation.cc:407] Connection header received
0619 13:05:38.892932 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:38.892939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:38.892986 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:38.893109 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:38.893975 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:38.893981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:38.893989 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:38.894009 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:38.894055 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:38.894081 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:38.894206 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:38.894296 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 13:05:38.894362 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:05:38.900432 22713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452b2b5aacd54e9da17d9f319504ae05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } }
I20260619 13:05:38.900805 22713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:38.900909 22713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452b2b5aacd54e9da17d9f319504ae05, State: Initialized, Role: FOLLOWER
I20260619 13:05:38.901520 22713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452b2b5aacd54e9da17d9f319504ae05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } }
I20260619 13:05:38.901811 22713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:38.901958 22713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:38.902134 22713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:38.904307 22713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452b2b5aacd54e9da17d9f319504ae05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } }
I20260619 13:05:38.904857 22713 leader_election.cc:302] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 452b2b5aacd54e9da17d9f319504ae05; no voters:
I20260619 13:05:38.905407 22713 leader_election.cc:288] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:38.905562 22723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:38.906212 22723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [term 1 LEADER]: Becoming Leader. State: Replica: 452b2b5aacd54e9da17d9f319504ae05, State: Running, Role: LEADER
I20260619 13:05:38.906759 22723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452b2b5aacd54e9da17d9f319504ae05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } }
I20260619 13:05:38.907407 22713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:38.909569 22724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "452b2b5aacd54e9da17d9f319504ae05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452b2b5aacd54e9da17d9f319504ae05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } } }
I20260619 13:05:38.910056 22724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:38.909559 22725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 452b2b5aacd54e9da17d9f319504ae05. Latest consensus state: current_term: 1 leader_uuid: "452b2b5aacd54e9da17d9f319504ae05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452b2b5aacd54e9da17d9f319504ae05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35647 } } }
I20260619 13:05:38.910681 22725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:38.911892 22730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:38.918184 22730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:38.929036 22730 catalog_manager.cc:1347] Generated new cluster ID: 0e97d4941c75453c932d3d5e18158ede
I20260619 13:05:38.929150 22730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:38.942636 22730 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:38.943125 22730 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:38.960002 22730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 452b2b5aacd54e9da17d9f319504ae05: Generated new TSK 0
I20260619 13:05:38.960903 22730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:38.976400 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22646
W20260619 13:05:38.977535 22708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.191:58075
W20260619 13:05:38.978197 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22646: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:38.978305 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 22646
2026-06-19T13:05:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:39 dist-test-slave-tpsm krb5kdc[22744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:39 dist-test-slave-tpsm krb5kdc[22744](info): set up 2 sockets
Jun 19 13:05:39 dist-test-slave-tpsm krb5kdc[22744](info): commencing operation
krb5kdc: starting...
W20260619 13:05:40.079751 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:40 dist-test-slave-tpsm krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:40.334736 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:40.661821 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:40.662178 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:40.662351 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:40.669566 22757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:40.669725 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:40.669823 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:40.669936 22757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:40.670027 22757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:40.679718 22757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:40.681025 22757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:40.682308 22757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:40.694010 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:40.694118 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:40.694052 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:40.694813 22757 server_base.cc:1034] running on GCE node
I20260619 13:05:40.695894 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:40.697582 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:40.698830 22757 hybrid_clock.cc:648] HybridClock initialized: now 1781874340698786 us; error 42 us; skew 500 ppm
I20260619 13:05:40.699198 22757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:40 dist-test-slave-tpsm krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:40.702344 22757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:40.703969 22757 webserver.cc:460] Webserver started at http://127.0.0.1:39339/ using document root <none> and password file <none>
I20260619 13:05:40.704694 22757 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:40.704850 22757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:40.705163 22757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:40.708632 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "b5f83e029e4446539ff29df529ca39c8"
format_stamp: "Formatted at 2026-06-19 13:05:40 on dist-test-slave-tpsm"
server_key: "bd8e15c52798ee036bb43cb816ffcf11"
server_key_iv: "9190207f0dc0ca00ed883acbfbb8c04d"
server_key_version: "encryptionkey@0"
I20260619 13:05:40.709638 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "b5f83e029e4446539ff29df529ca39c8"
format_stamp: "Formatted at 2026-06-19 13:05:40 on dist-test-slave-tpsm"
server_key: "bd8e15c52798ee036bb43cb816ffcf11"
server_key_iv: "9190207f0dc0ca00ed883acbfbb8c04d"
server_key_version: "encryptionkey@0"
I20260619 13:05:40.715647 22757 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 13:05:40.719368 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:40.720748 22757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:40.720938 22757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "b5f83e029e4446539ff29df529ca39c8"
format_stamp: "Formatted at 2026-06-19 13:05:40 on dist-test-slave-tpsm"
server_key: "bd8e15c52798ee036bb43cb816ffcf11"
server_key_iv: "9190207f0dc0ca00ed883acbfbb8c04d"
server_key_version: "encryptionkey@0"
I20260619 13:05:40.721143 22757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:40.733170 22757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:40.737324 22757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:40.737614 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:40.757091 22757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41225
I20260619 13:05:40.757181 22823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41225 every 8 connection(s)
I20260619 13:05:40.758530 22757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:40.758874 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 22757
I20260619 13:05:40.759099 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:40.759825 1373 external_mini_cluster.cc:1428] Setting key 97a43fef0db2c429419e16923cd5e53b
I20260619 13:05:40.763500 22824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:05:40 dist-test-slave-tpsm krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:05:40.773801 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8: Bootstrap starting.
I20260619 13:05:40.777010 22824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:40.778146 22824 log.cc:824] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:40.781328 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8: No bootstrap required, opened a new log
I20260619 13:05:40.781993 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:40.761728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:56283 (local address 127.0.0.1:41225)
0619 13:05:40.762460 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:40.762471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:40.762509 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:05:40.764297 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:40.764317 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:40.764674 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:40.765030 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:40.766448 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:40.767847 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:40.769088 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:40.769743 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:40.769941 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:40.774434 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:40.774441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:40.774454 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:40.774497 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:40.777648 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:40.778547 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:40.778553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:40.778558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:40.778792 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:40.779268 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:40.779274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:40.779283 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:40.779856 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:40.780090 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:40.780633 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 13:05:40.780919 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":302,"threads_started":1}
I20260619 13:05:40.785949 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:40.791501 22824 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f83e029e4446539ff29df529ca39c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260619 13:05:40.791975 22824 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:40.792081 22824 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f83e029e4446539ff29df529ca39c8, State: Initialized, Role: FOLLOWER
I20260619 13:05:40.792732 22824 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f83e029e4446539ff29df529ca39c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260619 13:05:40.792961 22824 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:40.793108 22824 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:40.793289 22824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:40.795578 22824 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f83e029e4446539ff29df529ca39c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260619 13:05:40.796149 22824 leader_election.cc:302] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f83e029e4446539ff29df529ca39c8; no voters:
I20260619 13:05:40.796821 22824 leader_election.cc:288] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:40.796957 22834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:40.798825 22824 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:40.801476 22834 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [term 1 LEADER]: Becoming Leader. State: Replica: b5f83e029e4446539ff29df529ca39c8, State: Running, Role: LEADER
I20260619 13:05:40.802254 22834 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f83e029e4446539ff29df529ca39c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260619 13:05:40.805617 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f83e029e4446539ff29df529ca39c8. Latest consensus state: current_term: 1 leader_uuid: "b5f83e029e4446539ff29df529ca39c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f83e029e4446539ff29df529ca39c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } } }
I20260619 13:05:40.806880 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:40.806110 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f83e029e4446539ff29df529ca39c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f83e029e4446539ff29df529ca39c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } } }
I20260619 13:05:40.807219 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:40.809638 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:40.791158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:60671 (local address 127.0.0.1:41225)
0619 13:05:40.791436 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:40.791441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:40.791759 (+ 318us) server_negotiation.cc:407] Connection header received
0619 13:05:40.791945 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:40.791958 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:40.792021 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:40.792162 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:40.793722 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:40.794559 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:40.795885 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:40.796556 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:40.796716 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:40.800570 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:40.800579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:40.800601 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:40.800660 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:40.804936 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:40.805808 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:40.805815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:40.805819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:40.806348 (+ 529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:40.807154 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:40.807162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:40.807231 (+ 69us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:40.807783 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:40.808231 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:40.808934 (+ 703us) server_negotiation.cc:299] Negotiation successful
0619 13:05:40.809168 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 13:05:40.814210 22849 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:05:40.814335 22849 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:05:40.815866 22848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:40.817576 22848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:40.825688 22848 catalog_manager.cc:1347] Generated new cluster ID: 6e8c75439bf74b088eb3a7ca28d0d5c6
I20260619 13:05:40.825809 22848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:40.836621 22848 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:40.837893 22848 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:40.847122 22848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5f83e029e4446539ff29df529ca39c8: Generated new TSK 0
I20260619 13:05:40.848143 22848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:40.860852 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22757
I20260619 13:05:40.861953 22819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:40.951764 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 22757
W20260619 13:05:40.968924 22742 connection.cc:535] client connection to 127.0.0.1:41225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41225): Connection reset by peer (error 104)
2026-06-19T13:05:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:41 dist-test-slave-tpsm krb5kdc[22856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:41 dist-test-slave-tpsm krb5kdc[22856](info): set up 2 sockets
Jun 19 13:05:41 dist-test-slave-tpsm krb5kdc[22856](info): commencing operation
krb5kdc: starting...
W20260619 13:05:42.059886 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:42 dist-test-slave-tpsm krb5kdc[22856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:42.322757 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:42.648901 22869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:42.649184 22869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:42.649371 22869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:42.656594 22869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:42.656733 22869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:42.656848 22869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:42.656960 22869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:42.657054 22869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:42.666997 22869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39123
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:42.668349 22869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:42.669812 22869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:42.681497 22874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:42.681708 22877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:42.681777 22875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:42.682775 22869 server_base.cc:1034] running on GCE node
I20260619 13:05:42.683670 22869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:42.685031 22869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:42.686255 22869 hybrid_clock.cc:648] HybridClock initialized: now 1781874342686206 us; error 31 us; skew 500 ppm
I20260619 13:05:42.686610 22869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:42 dist-test-slave-tpsm krb5kdc[22856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:42.689734 22869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:42.691145 22869 webserver.cc:460] Webserver started at http://127.0.0.1:46513/ using document root <none> and password file <none>
I20260619 13:05:42.691862 22869 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:42.691998 22869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:42.692312 22869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:42.695235 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "63df7c4dc0de4a39a77bdbc94dd27785"
format_stamp: "Formatted at 2026-06-19 13:05:42 on dist-test-slave-tpsm"
server_key: "de4cbaa323f62794f3a7d561079e7707"
server_key_iv: "72794398accee2264f58a1ae026460d7"
server_key_version: "encryptionkey@0"
I20260619 13:05:42.696066 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "63df7c4dc0de4a39a77bdbc94dd27785"
format_stamp: "Formatted at 2026-06-19 13:05:42 on dist-test-slave-tpsm"
server_key: "de4cbaa323f62794f3a7d561079e7707"
server_key_iv: "72794398accee2264f58a1ae026460d7"
server_key_version: "encryptionkey@0"
I20260619 13:05:42.700981 22869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:05:42.704331 22884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:42.705642 22869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:42.705926 22869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "63df7c4dc0de4a39a77bdbc94dd27785"
format_stamp: "Formatted at 2026-06-19 13:05:42 on dist-test-slave-tpsm"
server_key: "de4cbaa323f62794f3a7d561079e7707"
server_key_iv: "72794398accee2264f58a1ae026460d7"
server_key_version: "encryptionkey@0"
I20260619 13:05:42.706130 22869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:42.741707 22869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:42.746161 22869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:42.746491 22869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:42.766985 22869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39123
I20260619 13:05:42.767016 22935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39123 every 8 connection(s)
I20260619 13:05:42.768414 22869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:42.772274 22936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:42.778012 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 22869
I20260619 13:05:42.778198 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:42.778854 1373 external_mini_cluster.cc:1428] Setting key f466908909dc0dbed98dff4b2db45d2d
I20260619 13:05:42.784097 22936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785: Bootstrap starting.
I20260619 13:05:42.787096 22936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:42.788208 22936 log.cc:824] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:42.790946 22936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785: No bootstrap required, opened a new log
I20260619 13:05:42.791919 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:42.780870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:47833 (local address 127.0.0.1:39123)
0619 13:05:42.781702 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:42.781713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:42.781756 (+ 43us) server_negotiation.cc:407] Connection header received
0619 13:05:42.783497 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:42.783519 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:42.783904 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:42.784273 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:42.784820 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:42.786084 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:42.787536 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:42.788264 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:42.788447 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:42.789697 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:42.789704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:42.789717 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:42.790218 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:42.790288 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:42.790312 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:42.790483 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:42.790759 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 13:05:42.790913 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":337,"threads_started":1}
I20260619 13:05:42.795794 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:42.798877 22936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63df7c4dc0de4a39a77bdbc94dd27785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39123 } }
I20260619 13:05:42.799229 22936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:42.799381 22936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63df7c4dc0de4a39a77bdbc94dd27785, State: Initialized, Role: FOLLOWER
I20260619 13:05:42.799988 22936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63df7c4dc0de4a39a77bdbc94dd27785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39123 } }
I20260619 13:05:42.800411 22936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:42.800563 22936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:42.800740 22936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:42.802956 22936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63df7c4dc0de4a39a77bdbc94dd27785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39123 } }
I20260619 13:05:42.803627 22936 leader_election.cc:302] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63df7c4dc0de4a39a77bdbc94dd27785; no voters:
I20260619 13:05:42.804342 22936 leader_election.cc:288] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:42.804445 22946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:42.806871 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:42.800189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:49065 (local address 127.0.0.1:39123)
0619 13:05:42.800427 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:42.800435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:42.800830 (+ 395us) server_negotiation.cc:407] Connection header received
0619 13:05:42.801038 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:42.801042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:42.801099 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:42.801200 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:42.801739 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:42.802506 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:42.804014 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:42.804633 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:42.805444 (+ 811us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:42.806076 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:42.806081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:42.806084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:42.806100 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:42.806154 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:42.806168 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:42.806292 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:42.806479 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 13:05:42.806533 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:05:42.806914 22936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:42.807610 22946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [term 1 LEADER]: Becoming Leader. State: Replica: 63df7c4dc0de4a39a77bdbc94dd27785, State: Running, Role: LEADER
I20260619 13:05:42.808323 22946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63df7c4dc0de4a39a77bdbc94dd27785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39123 } }
I20260619 13:05:42.811544 22947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63df7c4dc0de4a39a77bdbc94dd27785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63df7c4dc0de4a39a77bdbc94dd27785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39123 } } }
I20260619 13:05:42.811987 22947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:42.812686 22948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63df7c4dc0de4a39a77bdbc94dd27785. Latest consensus state: current_term: 1 leader_uuid: "63df7c4dc0de4a39a77bdbc94dd27785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63df7c4dc0de4a39a77bdbc94dd27785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39123 } } }
I20260619 13:05:42.813110 22948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:42.821966 22953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:42.828307 22953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:42.840683 22953 catalog_manager.cc:1347] Generated new cluster ID: 10f1870d04d648639215eb508b2f8dd6
I20260619 13:05:42.840811 22953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:42.865358 22953 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:42.866693 22953 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:42.882172 22953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63df7c4dc0de4a39a77bdbc94dd27785: Generated new TSK 0
I20260619 13:05:42.882930 22953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:42.891103 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22869
W20260619 13:05:42.892202 22931 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.191:47833
W20260619 13:05:42.892894 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22869: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:42.893020 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 22869
W20260619 13:05:42.909767 22854 connection.cc:535] client connection to 127.0.0.1:39123 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39123): Connection reset by peer (error 104)
2026-06-19T13:05:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:42 dist-test-slave-tpsm krb5kdc[22967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:42 dist-test-slave-tpsm krb5kdc[22967](info): set up 2 sockets
Jun 19 13:05:42 dist-test-slave-tpsm krb5kdc[22967](info): commencing operation
krb5kdc: starting...
W20260619 13:05:44.011945 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:44 dist-test-slave-tpsm krb5kdc[22967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:05:44.266527 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:44.590600 22980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:44.590886 22980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:44.591046 22980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:44.598596 22980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:44.598706 22980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:44.598784 22980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:44.598879 22980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:44.598944 22980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:44.609022 22980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:44.610414 22980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:44.611726 22980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:44.623580 22985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:44.623540 22986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:44.623637 22988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:44.624531 22980 server_base.cc:1034] running on GCE node
I20260619 13:05:44.625360 22980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:44.626827 22980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:44.628046 22980 hybrid_clock.cc:648] HybridClock initialized: now 1781874344627984 us; error 59 us; skew 500 ppm
I20260619 13:05:44.628448 22980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:44 dist-test-slave-tpsm krb5kdc[22967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:44.634939 22980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:05:44.636658 22980 webserver.cc:460] Webserver started at http://127.0.0.1:40141/ using document root <none> and password file <none>
I20260619 13:05:44.637352 22980 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:44.637501 22980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:44.637810 22980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:44.640869 22980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "5d65e36980e54c84947dc901faef9ebd"
format_stamp: "Formatted at 2026-06-19 13:05:44 on dist-test-slave-tpsm"
server_key: "ff1a3291653500ecfa7d78f9d9690107"
server_key_iv: "2cb31c904e3673c8a3c31c1917d28f8d"
server_key_version: "encryptionkey@0"
I20260619 13:05:44.641626 22980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "5d65e36980e54c84947dc901faef9ebd"
format_stamp: "Formatted at 2026-06-19 13:05:44 on dist-test-slave-tpsm"
server_key: "ff1a3291653500ecfa7d78f9d9690107"
server_key_iv: "2cb31c904e3673c8a3c31c1917d28f8d"
server_key_version: "encryptionkey@0"
I20260619 13:05:44.646986 22980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:05:44.650418 22995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:44.651765 22980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:44.651948 22980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "5d65e36980e54c84947dc901faef9ebd"
format_stamp: "Formatted at 2026-06-19 13:05:44 on dist-test-slave-tpsm"
server_key: "ff1a3291653500ecfa7d78f9d9690107"
server_key_iv: "2cb31c904e3673c8a3c31c1917d28f8d"
server_key_version: "encryptionkey@0"
I20260619 13:05:44.652146 22980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:44.657815 22980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:44.662066 22980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:44.662366 22980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:44.681116 22980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45711
I20260619 13:05:44.681169 23046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45711 every 8 connection(s)
I20260619 13:05:44.682488 22980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:44.686038 23047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:44.690582 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 22980
I20260619 13:05:44.690797 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:44.691464 1373 external_mini_cluster.cc:1428] Setting key d53018bb4f1f2ac6d05752d3f3432b2d
I20260619 13:05:44.696465 23047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd: Bootstrap starting.
I20260619 13:05:44.699553 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:44.693381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:46453 (local address 127.0.0.1:45711)
0619 13:05:44.694301 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:44.694317 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:44.694372 (+ 55us) server_negotiation.cc:407] Connection header received
0619 13:05:44.696174 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:44.696195 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:44.696513 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:44.696903 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:44.697932 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:44.697938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:44.697955 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:44.697970 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:44.698044 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:44.698066 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:44.698249 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:44.698355 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 13:05:44.698578 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":421,"threads_started":1}
I20260619 13:05:44.699993 23047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:44.701016 23047 log.cc:824] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:44.703517 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:44.704082 23047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd: No bootstrap required, opened a new log
I20260619 13:05:44.711015 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:44.708092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.191:40809 (local address 127.0.0.1:45711)
0619 13:05:44.708342 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:44.708352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:44.708853 (+ 501us) server_negotiation.cc:407] Connection header received
0619 13:05:44.709078 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:44.709087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:44.709137 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:44.709260 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:44.710244 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:44.710250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:44.710259 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:44.710279 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:44.710328 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:44.710348 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:44.710488 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:44.710653 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 13:05:44.710723 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:05:44.711786 23047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d65e36980e54c84947dc901faef9ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45711 } }
I20260619 13:05:44.712157 23047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:44.712277 23047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d65e36980e54c84947dc901faef9ebd, State: Initialized, Role: FOLLOWER
I20260619 13:05:44.712826 23047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d65e36980e54c84947dc901faef9ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45711 } }
I20260619 13:05:44.713076 23047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:44.713214 23047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:44.713403 23047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:44.715728 23047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d65e36980e54c84947dc901faef9ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45711 } }
I20260619 13:05:44.716207 23047 leader_election.cc:302] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d65e36980e54c84947dc901faef9ebd; no voters:
I20260619 13:05:44.716842 23047 leader_election.cc:288] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:44.717010 23057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:44.717685 23057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [term 1 LEADER]: Becoming Leader. State: Replica: 5d65e36980e54c84947dc901faef9ebd, State: Running, Role: LEADER
I20260619 13:05:44.718253 23057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d65e36980e54c84947dc901faef9ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45711 } }
I20260619 13:05:44.718947 23047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:44.720988 23058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d65e36980e54c84947dc901faef9ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d65e36980e54c84947dc901faef9ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45711 } } }
I20260619 13:05:44.721316 23058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:44.722064 23059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d65e36980e54c84947dc901faef9ebd. Latest consensus state: current_term: 1 leader_uuid: "5d65e36980e54c84947dc901faef9ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d65e36980e54c84947dc901faef9ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45711 } } }
I20260619 13:05:44.722452 23059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:44.723055 23062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:44.729993 23062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:44.737649 23062 catalog_manager.cc:1347] Generated new cluster ID: e96820bfe2574c08b79186dadb06765d
I20260619 13:05:44.737753 23062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:44.749778 23062 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:44.750219 23062 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:44.761442 23062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d65e36980e54c84947dc901faef9ebd: Generated new TSK 0
I20260619 13:05:44.762494 23062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:44.775911 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22980
W20260619 13:05:44.777499 23042 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.191:46453
W20260619 13:05:44.778448 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 22980: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:44.778625 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 22980
2026-06-19T13:05:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:44 dist-test-slave-tpsm krb5kdc[23078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:44 dist-test-slave-tpsm krb5kdc[23078](info): set up 2 sockets
Jun 19 13:05:44 dist-test-slave-tpsm krb5kdc[23078](info): commencing operation
krb5kdc: starting...
W20260619 13:05:45.875839 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:46 dist-test-slave-tpsm krb5kdc[23078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:46.132726 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:38447
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:38447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:46.454594 23091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:46.454869 23091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:46.455071 23091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:46.462178 23091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:46.462275 23091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:46.462387 23091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:46.462476 23091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:46.462563 23091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:46.474793 23091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:38447
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:38447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:46.476670 23091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:46.478426 23091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:46.489807 23096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:46.490033 23097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:46.492296 23091 server_base.cc:1034] running on GCE node
W20260619 13:05:46.492503 23099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:46.493255 23091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:46.494616 23091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:46.495862 23091 hybrid_clock.cc:648] HybridClock initialized: now 1781874346495819 us; error 47 us; skew 500 ppm
I20260619 13:05:46.496249 23091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:46 dist-test-slave-tpsm krb5kdc[23078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:46.499285 23091 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:46.500896 23091 webserver.cc:460] Webserver started at http://127.1.87.126:36199/ using document root <none> and password file <none>
I20260619 13:05:46.501590 23091 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:46.501735 23091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:46.502034 23091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:46.504981 23091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2"
format_stamp: "Formatted at 2026-06-19 13:05:46 on dist-test-slave-tpsm"
server_key: "4d0c97c0acd49d7f80d4d9dded8f72e7"
server_key_iv: "fbad82bf168605fffb2402a8ddc4e2de"
server_key_version: "encryptionkey@0"
I20260619 13:05:46.505705 23091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2"
format_stamp: "Formatted at 2026-06-19 13:05:46 on dist-test-slave-tpsm"
server_key: "4d0c97c0acd49d7f80d4d9dded8f72e7"
server_key_iv: "fbad82bf168605fffb2402a8ddc4e2de"
server_key_version: "encryptionkey@0"
I20260619 13:05:46.510547 23091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260619 13:05:46.513880 23106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:46.515098 23091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:05:46.515288 23091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2"
format_stamp: "Formatted at 2026-06-19 13:05:46 on dist-test-slave-tpsm"
server_key: "4d0c97c0acd49d7f80d4d9dded8f72e7"
server_key_iv: "fbad82bf168605fffb2402a8ddc4e2de"
server_key_version: "encryptionkey@0"
I20260619 13:05:46.515542 23091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:46.542460 23091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:46.546763 23091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:46.547048 23091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:46.565418 23091 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:38447
I20260619 13:05:46.565452 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:38447 every 8 connection(s)
I20260619 13:05:46.566744 23091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:46.570328 23158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:46.576843 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 23091
I20260619 13:05:46.577024 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:46.577540 1373 external_mini_cluster.cc:1428] Setting key 6726bdea86feb755aafef3f7c7a558cd
I20260619 13:05:46.581180 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2: Bootstrap starting.
I20260619 13:05:46.584158 23158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:46.585188 23158 log.cc:824] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:46.588114 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2: No bootstrap required, opened a new log
Jun 19 13:05:46 dist-test-slave-tpsm krb5kdc[23078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:46.596728 23158 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38447 } }
I20260619 13:05:46.597107 23158 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:46.597257 23158 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4893bc94aa4f26bcdefdc8667ac9f2, State: Initialized, Role: FOLLOWER
I20260619 13:05:46.597849 23158 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38447 } }
I20260619 13:05:46.598102 23158 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:46.598260 23158 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:46.598439 23158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:46.600636 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:46.579428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34021 (local address 127.1.87.126:38447)
0619 13:05:46.580214 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:46.580225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:46.580264 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:05:46.582030 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:46.582051 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:46.582416 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:46.582806 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:46.584224 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:46.585637 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:46.587236 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:46.587940 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:46.588073 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:46.592866 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:46.592874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:46.592887 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:46.592927 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:46.596354 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:46.597115 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:46.597122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:46.597127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:46.597375 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:46.597833 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:46.597841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:46.597845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:46.598493 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:46.598837 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:46.599351 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 13:05:46.599673 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":403,"threads_started":1}
I20260619 13:05:46.601155 23158 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38447 } }
I20260619 13:05:46.601702 23158 leader_election.cc:302] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa4893bc94aa4f26bcdefdc8667ac9f2; no voters:
I20260619 13:05:46.602393 23158 leader_election.cc:288] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:46.602542 23163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:46.604007 23163 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [term 1 LEADER]: Becoming Leader. State: Replica: fa4893bc94aa4f26bcdefdc8667ac9f2, State: Running, Role: LEADER
I20260619 13:05:46.604585 23158 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:46.604626 23163 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38447 } }
I20260619 13:05:46.605064 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:46.611883 23165 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa4893bc94aa4f26bcdefdc8667ac9f2. Latest consensus state: current_term: 1 leader_uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38447 } } }
I20260619 13:05:46.612337 23165 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:46.613265 23176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:46.613135 23164 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4893bc94aa4f26bcdefdc8667ac9f2" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38447 } } }
I20260619 13:05:46.613464 23164 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:46.627970 23176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:46.631578 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:46.611673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39425 (local address 127.1.87.126:38447)
0619 13:05:46.611844 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:46.611849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:46.611874 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:05:46.611931 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:46.611936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:46.612013 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:46.612128 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:46.614017 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:46.615357 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:46.617762 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:46.618670 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:46.618839 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:46.622584 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:46.622593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:46.622597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:46.622633 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:46.627018 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:46.628202 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:46.628207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:46.628224 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:46.628373 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:46.628946 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:46.628951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:46.628955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:46.629414 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:46.629693 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:46.630872 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0619 13:05:46.631059 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:05:46.638990 23176 catalog_manager.cc:1347] Generated new cluster ID: f5b3ca09e7474a50a387ed8ad040c303
I20260619 13:05:46.639199 23176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:46.671136 23176 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:46.672873 23176 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:46.681777 23176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa4893bc94aa4f26bcdefdc8667ac9f2: Generated new TSK 0
I20260619 13:05:46.682814 23176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:46.699021 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23091
I20260619 13:05:46.700343 23153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:46.795812 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 23091
W20260619 13:05:46.813301 23076 connection.cc:535] client connection to 127.1.87.126:38447 recv error: Network error: failed to read from TLS socket (remote: 127.1.87.126:38447): Connection reset by peer (error 104)
2026-06-19T13:05:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:46 dist-test-slave-tpsm krb5kdc[23190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:46 dist-test-slave-tpsm krb5kdc[23190](info): set up 2 sockets
Jun 19 13:05:46 dist-test-slave-tpsm krb5kdc[23190](info): commencing operation
krb5kdc: starting...
W20260619 13:05:47.896034 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:48 dist-test-slave-tpsm krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:48.156250 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:42285
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:42285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:48.480135 23203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:48.480422 23203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:48.480572 23203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:48.487816 23203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:48.487926 23203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:48.488044 23203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:48.488125 23203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:48.488188 23203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:48.498023 23203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:42285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:42285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:48.499269 23203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:48.500572 23203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:48.511935 23211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:48.511974 23208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:48.511896 23209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:48.513123 23203 server_base.cc:1034] running on GCE node
I20260619 13:05:48.513787 23203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:48.515043 23203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:48.516234 23203 hybrid_clock.cc:648] HybridClock initialized: now 1781874348516193 us; error 42 us; skew 500 ppm
I20260619 13:05:48.516592 23203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:48 dist-test-slave-tpsm krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874348, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:48.519711 23203 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:48.521277 23203 webserver.cc:460] Webserver started at http://127.1.87.126:38581/ using document root <none> and password file <none>
I20260619 13:05:48.521963 23203 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:48.522110 23203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:48.522426 23203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:48.525902 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4"
format_stamp: "Formatted at 2026-06-19 13:05:48 on dist-test-slave-tpsm"
server_key: "09e408bf2f4b837e7bd939958a2f2e23"
server_key_iv: "4841f597f157b497bf03e1df1158bcbe"
server_key_version: "encryptionkey@0"
I20260619 13:05:48.526862 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4"
format_stamp: "Formatted at 2026-06-19 13:05:48 on dist-test-slave-tpsm"
server_key: "09e408bf2f4b837e7bd939958a2f2e23"
server_key_iv: "4841f597f157b497bf03e1df1158bcbe"
server_key_version: "encryptionkey@0"
I20260619 13:05:48.532265 23203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:05:48.535717 23218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:48.537050 23203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:05:48.537326 23203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4"
format_stamp: "Formatted at 2026-06-19 13:05:48 on dist-test-slave-tpsm"
server_key: "09e408bf2f4b837e7bd939958a2f2e23"
server_key_iv: "4841f597f157b497bf03e1df1158bcbe"
server_key_version: "encryptionkey@0"
I20260619 13:05:48.537528 23203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:48.550493 23203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:48.554523 23203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:48.554812 23203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:48.573045 23203 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:42285
I20260619 13:05:48.573077 23269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:42285 every 8 connection(s)
I20260619 13:05:48.574419 23203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:48.577981 23270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:48.580910 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 23203
I20260619 13:05:48.581095 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:48.581641 1373 external_mini_cluster.cc:1428] Setting key 23ce22950561a95451f313bfa0050409
I20260619 13:05:48.589604 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4: Bootstrap starting.
I20260619 13:05:48.593057 23270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:48.594193 23270 log.cc:824] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:48.594326 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:48.583411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46947 (local address 127.1.87.126:42285)
0619 13:05:48.584146 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:48.584157 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:48.584194 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:05:48.585825 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:48.585843 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:48.586197 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:48.586503 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:48.587056 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:48.588397 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:48.590156 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:48.590764 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:48.590886 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:48.592218 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:48.592225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:48.592236 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:48.592723 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:48.592793 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:48.592815 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:48.592924 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:48.593294 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 13:05:48.593422 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":343,"threads_started":1}
I20260619 13:05:48.597117 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4: No bootstrap required, opened a new log
I20260619 13:05:48.597901 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:48.605238 23270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42285 } }
I20260619 13:05:48.605651 23270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:48.605798 23270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aee2d2f34c44ce7909a4bb3a0e878d4, State: Initialized, Role: FOLLOWER
I20260619 13:05:48.606438 23270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42285 } }
I20260619 13:05:48.606698 23270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:48.606848 23270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:48.607048 23270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:48.608317 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:48.602132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54793 (local address 127.1.87.126:42285)
0619 13:05:48.602322 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:48.602330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:48.602673 (+ 343us) server_negotiation.cc:407] Connection header received
0619 13:05:48.602886 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:48.602890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:48.602951 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:48.603068 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:48.603573 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:48.604339 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:48.605655 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:48.606194 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:48.606338 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:48.607573 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:48.607580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:48.607589 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:48.607606 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:48.607665 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:48.607681 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:48.607818 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:48.607983 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 13:05:48.608039 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:05:48.610165 23270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42285 } }
I20260619 13:05:48.610724 23270 leader_election.cc:302] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aee2d2f34c44ce7909a4bb3a0e878d4; no voters:
I20260619 13:05:48.611394 23270 leader_election.cc:288] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:48.611523 23280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:48.612277 23280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [term 1 LEADER]: Becoming Leader. State: Replica: 4aee2d2f34c44ce7909a4bb3a0e878d4, State: Running, Role: LEADER
I20260619 13:05:48.612942 23280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42285 } }
I20260619 13:05:48.613725 23270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:48.616026 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42285 } } }
I20260619 13:05:48.616123 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aee2d2f34c44ce7909a4bb3a0e878d4. Latest consensus state: current_term: 1 leader_uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aee2d2f34c44ce7909a4bb3a0e878d4" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 42285 } } }
I20260619 13:05:48.616410 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:48.616410 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:48.623799 23287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:48.629551 23287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:48.638489 23287 catalog_manager.cc:1347] Generated new cluster ID: 9f75750a975648f692f84c7508ccf9d8
I20260619 13:05:48.638622 23287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:48.656137 23287 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:48.658079 23287 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:48.684695 23287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aee2d2f34c44ce7909a4bb3a0e878d4: Generated new TSK 0
I20260619 13:05:48.685616 23287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:48.700028 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23203
W20260619 13:05:48.701352 23265 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46947
W20260619 13:05:48.702114 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:48.702275 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 23203
W20260619 13:05:48.720650 23188 connection.cc:535] client connection to 127.1.87.126:42285 recv error: Network error: failed to read from TLS socket (remote: 127.1.87.126:42285): Connection reset by peer (error 104)
2026-06-19T13:05:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:48 dist-test-slave-tpsm krb5kdc[23301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:48 dist-test-slave-tpsm krb5kdc[23301](info): set up 2 sockets
Jun 19 13:05:48 dist-test-slave-tpsm krb5kdc[23301](info): commencing operation
krb5kdc: starting...
W20260619 13:05:49.811906 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:49 dist-test-slave-tpsm krb5kdc[23301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:50.065080 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39421
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:39421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:50.402585 23314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:50.402868 23314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:50.403038 23314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:50.410207 23314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:50.410322 23314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:50.410413 23314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:50.410506 23314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:50.410574 23314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:50.420178 23314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:39421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:50.421443 23314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:50.422698 23314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:50.436195 23322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:50.436281 23320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:50.436275 23319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:50.437381 23314 server_base.cc:1034] running on GCE node
I20260619 13:05:50.438267 23314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:50.439774 23314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:50.441010 23314 hybrid_clock.cc:648] HybridClock initialized: now 1781874350440986 us; error 51 us; skew 500 ppm
I20260619 13:05:50.441372 23314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:50 dist-test-slave-tpsm krb5kdc[23301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874350, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:50.447144 23314 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:50.448685 23314 webserver.cc:460] Webserver started at http://127.1.87.126:44857/ using document root <none> and password file <none>
I20260619 13:05:50.449388 23314 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:50.449536 23314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:50.449847 23314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:50.452867 23314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "84ef7c2d2aa94cb3a28e6d27b916de43"
format_stamp: "Formatted at 2026-06-19 13:05:50 on dist-test-slave-tpsm"
server_key: "fa851d31ab9e3da713fe25b99ac8f3ff"
server_key_iv: "ac22731d588d34f3f4dc00b506e75285"
server_key_version: "encryptionkey@0"
I20260619 13:05:50.453613 23314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "84ef7c2d2aa94cb3a28e6d27b916de43"
format_stamp: "Formatted at 2026-06-19 13:05:50 on dist-test-slave-tpsm"
server_key: "fa851d31ab9e3da713fe25b99ac8f3ff"
server_key_iv: "ac22731d588d34f3f4dc00b506e75285"
server_key_version: "encryptionkey@0"
I20260619 13:05:50.459043 23314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 13:05:50.462510 23329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:50.463799 23314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:50.463985 23314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "84ef7c2d2aa94cb3a28e6d27b916de43"
format_stamp: "Formatted at 2026-06-19 13:05:50 on dist-test-slave-tpsm"
server_key: "fa851d31ab9e3da713fe25b99ac8f3ff"
server_key_iv: "ac22731d588d34f3f4dc00b506e75285"
server_key_version: "encryptionkey@0"
I20260619 13:05:50.464193 23314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:50.471624 23314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:50.477101 23314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:50.477413 23314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:50.494494 23314 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:39421
I20260619 13:05:50.494529 23380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:39421 every 8 connection(s)
I20260619 13:05:50.495991 23314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:50.499269 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 23314
I20260619 13:05:50.499485 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:50.499774 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:50.499987 1373 external_mini_cluster.cc:1428] Setting key d0af371b81b4178d39d40f93b0e2d9d5
I20260619 13:05:50.508456 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:50.501735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48725 (local address 127.1.87.126:39421)
0619 13:05:50.502526 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:50.502541 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:50.502591 (+ 50us) server_negotiation.cc:407] Connection header received
0619 13:05:50.504720 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:50.504746 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:50.505149 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:50.505578 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:50.506517 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:50.506525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:50.506540 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:50.506556 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:50.506625 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:50.506657 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:50.506862 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:50.507046 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 13:05:50.507250 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":304,"threads_started":1}
I20260619 13:05:50.511977 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:50.514135 23381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43: Bootstrap starting.
I20260619 13:05:50.517691 23381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:50.519332 23381 log.cc:824] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:50.519590 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:50.516207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53417 (local address 127.1.87.126:39421)
0619 13:05:50.516436 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:50.516442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:50.516822 (+ 380us) server_negotiation.cc:407] Connection header received
0619 13:05:50.516981 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:50.516984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:50.517032 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:50.517109 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:50.518192 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:50.518201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:50.518205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:50.518223 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:50.518271 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:50.518288 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:50.518418 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:50.519109 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 13:05:50.519193 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:05:50.522956 23381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43: No bootstrap required, opened a new log
I20260619 13:05:50.531965 23381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ef7c2d2aa94cb3a28e6d27b916de43" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39421 } }
I20260619 13:05:50.532446 23381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:50.532616 23381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ef7c2d2aa94cb3a28e6d27b916de43, State: Initialized, Role: FOLLOWER
I20260619 13:05:50.533356 23381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ef7c2d2aa94cb3a28e6d27b916de43" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39421 } }
I20260619 13:05:50.533634 23381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:50.533797 23381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:50.533980 23381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:50.536306 23381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ef7c2d2aa94cb3a28e6d27b916de43" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39421 } }
I20260619 13:05:50.536839 23381 leader_election.cc:302] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84ef7c2d2aa94cb3a28e6d27b916de43; no voters:
I20260619 13:05:50.537501 23381 leader_election.cc:288] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:50.537636 23391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:50.538367 23391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [term 1 LEADER]: Becoming Leader. State: Replica: 84ef7c2d2aa94cb3a28e6d27b916de43, State: Running, Role: LEADER
I20260619 13:05:50.539127 23391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ef7c2d2aa94cb3a28e6d27b916de43" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39421 } }
I20260619 13:05:50.540105 23381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:50.542438 23392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84ef7c2d2aa94cb3a28e6d27b916de43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ef7c2d2aa94cb3a28e6d27b916de43" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39421 } } }
I20260619 13:05:50.542943 23392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:50.542584 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84ef7c2d2aa94cb3a28e6d27b916de43. Latest consensus state: current_term: 1 leader_uuid: "84ef7c2d2aa94cb3a28e6d27b916de43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ef7c2d2aa94cb3a28e6d27b916de43" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 39421 } } }
I20260619 13:05:50.543273 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:50.544950 23400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:50.549998 23400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:50.559427 23400 catalog_manager.cc:1347] Generated new cluster ID: dcedcf6167504796a75009d156e6df73
I20260619 13:05:50.559594 23400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:50.598033 23400 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:50.598519 23400 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:50.610142 23400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84ef7c2d2aa94cb3a28e6d27b916de43: Generated new TSK 0
I20260619 13:05:50.611080 23400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:50.616082 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23314
W20260619 13:05:50.617658 23376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48725
W20260619 13:05:50.618463 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23314: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:50.618600 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 23314
2026-06-19T13:05:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:50 dist-test-slave-tpsm krb5kdc[23412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:50 dist-test-slave-tpsm krb5kdc[23412](info): set up 2 sockets
Jun 19 13:05:50 dist-test-slave-tpsm krb5kdc[23412](info): commencing operation
krb5kdc: starting...
W20260619 13:05:51.743949 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:51 dist-test-slave-tpsm krb5kdc[23412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:52.017827 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:34579
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:34579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:52.349017 23425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:52.349336 23425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:52.349493 23425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:52.357396 23425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:52.357504 23425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:52.357602 23425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:52.357689 23425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:52.357765 23425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:52.367853 23425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:34579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:34579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:52.369171 23425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:52.370474 23425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:52.382197 23433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:52.382366 23430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:52.382362 23431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:52.383621 23425 server_base.cc:1034] running on GCE node
I20260619 13:05:52.384356 23425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:52.385761 23425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:52.386963 23425 hybrid_clock.cc:648] HybridClock initialized: now 1781874352386916 us; error 36 us; skew 500 ppm
I20260619 13:05:52.387290 23425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:52 dist-test-slave-tpsm krb5kdc[23412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874352, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:52.390575 23425 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:52.392277 23425 webserver.cc:460] Webserver started at http://127.1.87.126:38913/ using document root <none> and password file <none>
I20260619 13:05:52.393182 23425 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:52.393347 23425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:52.393698 23425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:52.396916 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "4f1e84e27b1049ba825af06da34f52cb"
format_stamp: "Formatted at 2026-06-19 13:05:52 on dist-test-slave-tpsm"
server_key: "421896bc00b96192588899376e9bb05a"
server_key_iv: "be9801ea122e626573841ea9ba13002e"
server_key_version: "encryptionkey@0"
I20260619 13:05:52.397701 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "4f1e84e27b1049ba825af06da34f52cb"
format_stamp: "Formatted at 2026-06-19 13:05:52 on dist-test-slave-tpsm"
server_key: "421896bc00b96192588899376e9bb05a"
server_key_iv: "be9801ea122e626573841ea9ba13002e"
server_key_version: "encryptionkey@0"
I20260619 13:05:52.403609 23425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:05:52.407835 23440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:52.409340 23425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:05:52.409574 23425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "4f1e84e27b1049ba825af06da34f52cb"
format_stamp: "Formatted at 2026-06-19 13:05:52 on dist-test-slave-tpsm"
server_key: "421896bc00b96192588899376e9bb05a"
server_key_iv: "be9801ea122e626573841ea9ba13002e"
server_key_version: "encryptionkey@0"
I20260619 13:05:52.409811 23425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:52.432319 23425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:52.436837 23425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:52.437152 23425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:52.456873 23425 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:34579
I20260619 13:05:52.456930 23491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:34579 every 8 connection(s)
I20260619 13:05:52.458287 23425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:52.462018 23492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:52.464249 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 23425
I20260619 13:05:52.464440 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:52.464879 1373 external_mini_cluster.cc:1428] Setting key 6832bc962a934bb872a2b31d44b19a70
I20260619 13:05:52.476106 23492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb: Bootstrap starting.
I20260619 13:05:52.479805 23492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb: Neither blocks nor log segments found. Creating new log.
Jun 19 13:05:52 dist-test-slave-tpsm krb5kdc[23412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:52.481037 23492 log.cc:824] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:52.484645 23492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb: No bootstrap required, opened a new log
I20260619 13:05:52.490622 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:52.466748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54063 (local address 127.1.87.126:34579)
0619 13:05:52.467726 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:52.467738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:52.467807 (+ 69us) server_negotiation.cc:407] Connection header received
0619 13:05:52.470798 (+ 2991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:52.470829 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:52.471390 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:52.471865 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:52.473898 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:52.475363 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:52.476696 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:52.477367 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:52.477571 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:52.482139 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:52.482147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:52.482161 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:52.482202 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:52.486138 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:52.486810 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:52.486818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:52.486826 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:52.487033 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:52.487448 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:52.487453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:52.487457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:52.488186 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:52.488493 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:52.488779 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 13:05:52.489226 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":370,"threads_started":1}
I20260619 13:05:52.494521 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:52.495060 23492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1e84e27b1049ba825af06da34f52cb" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34579 } }
I20260619 13:05:52.495496 23492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:52.495596 23492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1e84e27b1049ba825af06da34f52cb, State: Initialized, Role: FOLLOWER
I20260619 13:05:52.496191 23492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1e84e27b1049ba825af06da34f52cb" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34579 } }
I20260619 13:05:52.496423 23492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:52.496572 23492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:52.496757 23492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:52.499210 23492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1e84e27b1049ba825af06da34f52cb" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34579 } }
I20260619 13:05:52.499809 23492 leader_election.cc:302] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f1e84e27b1049ba825af06da34f52cb; no voters:
I20260619 13:05:52.500456 23492 leader_election.cc:288] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:52.500584 23502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:52.502573 23492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:52.503038 23502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [term 1 LEADER]: Becoming Leader. State: Replica: 4f1e84e27b1049ba825af06da34f52cb, State: Running, Role: LEADER
I20260619 13:05:52.503644 23502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1e84e27b1049ba825af06da34f52cb" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34579 } }
I20260619 13:05:52.507256 23504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f1e84e27b1049ba825af06da34f52cb. Latest consensus state: current_term: 1 leader_uuid: "4f1e84e27b1049ba825af06da34f52cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1e84e27b1049ba825af06da34f52cb" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34579 } } }
I20260619 13:05:52.507766 23504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:52.513216 23503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f1e84e27b1049ba825af06da34f52cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1e84e27b1049ba825af06da34f52cb" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 34579 } } }
I20260619 13:05:52.514323 23503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:52.515712 23512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:52.519873 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:52.499247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33207 (local address 127.1.87.126:34579)
0619 13:05:52.499521 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:52.499530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:52.500116 (+ 586us) server_negotiation.cc:407] Connection header received
0619 13:05:52.500329 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:52.500338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:52.500428 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:52.500554 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:52.502784 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:52.503999 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:52.506045 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:52.506736 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:52.506878 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:52.510702 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:52.510712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:52.510720 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:52.510773 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:52.515450 (+ 4677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:52.516424 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:52.516429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:52.516432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:52.516597 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:52.517097 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:52.517102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:52.517105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:52.517572 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:52.517898 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:52.518987 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0619 13:05:52.519237 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 13:05:52.521368 23512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:52.531306 23512 catalog_manager.cc:1347] Generated new cluster ID: 0233ed38e25547ce885a10c77cef6843
I20260619 13:05:52.531481 23512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:52.542346 23512 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:52.544301 23512 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:52.563815 23512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f1e84e27b1049ba825af06da34f52cb: Generated new TSK 0
I20260619 13:05:52.564702 23512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:52.580054 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23425
I20260619 13:05:52.581110 23487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:05:52.673007 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 23425
2026-06-19T13:05:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2051 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:52 dist-test-slave-tpsm krb5kdc[23524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:52 dist-test-slave-tpsm krb5kdc[23524](info): set up 2 sockets
Jun 19 13:05:52 dist-test-slave-tpsm krb5kdc[23524](info): commencing operation
krb5kdc: starting...
W20260619 13:05:53.787906 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:53 dist-test-slave-tpsm krb5kdc[23524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:54.059427 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:38191
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:38191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:54.391626 23537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:54.391973 23537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:54.392144 23537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:54.399248 23537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:54.399441 23537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:54.399557 23537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:54.399672 23537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:54.399768 23537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:54.409686 23537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:38191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:38191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:54.410990 23537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:54.412312 23537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:54.424242 23542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:54.424191 23545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:54.424276 23543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:54.425098 23537 server_base.cc:1034] running on GCE node
I20260619 13:05:54.425871 23537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:54.427305 23537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:54.428565 23537 hybrid_clock.cc:648] HybridClock initialized: now 1781874354428518 us; error 43 us; skew 500 ppm
I20260619 13:05:54.428907 23537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:54 dist-test-slave-tpsm krb5kdc[23524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874354, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:54.431984 23537 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:54.433559 23537 webserver.cc:460] Webserver started at http://127.1.87.126:39019/ using document root <none> and password file <none>
I20260619 13:05:54.434170 23537 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:54.434276 23537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:54.434520 23537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:54.437388 23537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "b6329f9c8cd8476189f04ea3878e9abc"
format_stamp: "Formatted at 2026-06-19 13:05:54 on dist-test-slave-tpsm"
server_key: "75ae9aba9d4d721ca31c15eff932b36e"
server_key_iv: "ee7260634d41b4a812e5c5897feb7a43"
server_key_version: "encryptionkey@0"
I20260619 13:05:54.438083 23537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "b6329f9c8cd8476189f04ea3878e9abc"
format_stamp: "Formatted at 2026-06-19 13:05:54 on dist-test-slave-tpsm"
server_key: "75ae9aba9d4d721ca31c15eff932b36e"
server_key_iv: "ee7260634d41b4a812e5c5897feb7a43"
server_key_version: "encryptionkey@0"
I20260619 13:05:54.443280 23537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:05:54.446681 23552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:54.448001 23537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:54.448138 23537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "b6329f9c8cd8476189f04ea3878e9abc"
format_stamp: "Formatted at 2026-06-19 13:05:54 on dist-test-slave-tpsm"
server_key: "75ae9aba9d4d721ca31c15eff932b36e"
server_key_iv: "ee7260634d41b4a812e5c5897feb7a43"
server_key_version: "encryptionkey@0"
I20260619 13:05:54.448308 23537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:54.489101 23537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:54.493119 23537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:54.493363 23537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:54.511812 23537 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:38191
I20260619 13:05:54.511850 23603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:38191 every 8 connection(s)
I20260619 13:05:54.513164 23537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:54.514637 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 23537
I20260619 13:05:54.514827 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:54.515484 1373 external_mini_cluster.cc:1428] Setting key 5f84b090b767583689363fc5d3189944
I20260619 13:05:54.517774 23604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:54.528542 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:54.517271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54433 (local address 127.1.87.126:38191)
0619 13:05:54.518057 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:54.518068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:54.518106 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:05:54.519713 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:54.519734 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:54.520090 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:54.520408 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:54.520975 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:54.522297 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:54.524182 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:54.524815 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:54.524937 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:54.526258 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:54.526265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:54.526277 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:54.526830 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:54.526904 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:54.526930 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:54.527014 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:54.527270 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 13:05:54.527619 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":351,"threads_started":1}
I20260619 13:05:54.530306 23604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc: Bootstrap starting.
I20260619 13:05:54.532255 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:54.533679 23604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:54.534664 23604 log.cc:824] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:54.538365 23604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc: No bootstrap required, opened a new log
I20260619 13:05:54.544457 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:54.536501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40531 (local address 127.1.87.126:38191)
0619 13:05:54.536659 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:54.536663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:54.538972 (+ 2309us) server_negotiation.cc:407] Connection header received
0619 13:05:54.539190 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:54.539195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:54.539260 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:54.539393 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:54.539888 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:54.540641 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:54.541997 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:54.542459 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:54.542608 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:54.543708 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:54.543715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:54.543719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:54.543742 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:54.543781 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:54.543796 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:54.543901 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:54.544050 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 13:05:54.544140 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 13:05:54.546754 23604 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6329f9c8cd8476189f04ea3878e9abc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38191 } }
I20260619 13:05:54.547123 23604 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:54.547205 23604 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6329f9c8cd8476189f04ea3878e9abc, State: Initialized, Role: FOLLOWER
I20260619 13:05:54.547744 23604 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6329f9c8cd8476189f04ea3878e9abc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38191 } }
I20260619 13:05:54.547940 23604 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:54.548032 23604 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:54.548193 23604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:54.550655 23604 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6329f9c8cd8476189f04ea3878e9abc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38191 } }
I20260619 13:05:54.551128 23604 leader_election.cc:302] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6329f9c8cd8476189f04ea3878e9abc; no voters:
I20260619 13:05:54.551903 23604 leader_election.cc:288] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:54.552049 23614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:54.552872 23614 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [term 1 LEADER]: Becoming Leader. State: Replica: b6329f9c8cd8476189f04ea3878e9abc, State: Running, Role: LEADER
I20260619 13:05:54.553390 23614 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6329f9c8cd8476189f04ea3878e9abc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38191 } }
I20260619 13:05:54.554152 23604 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:54.556319 23616 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6329f9c8cd8476189f04ea3878e9abc. Latest consensus state: current_term: 1 leader_uuid: "b6329f9c8cd8476189f04ea3878e9abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6329f9c8cd8476189f04ea3878e9abc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38191 } } }
I20260619 13:05:54.556681 23616 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:54.557490 23615 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6329f9c8cd8476189f04ea3878e9abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6329f9c8cd8476189f04ea3878e9abc" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 38191 } } }
I20260619 13:05:54.557865 23615 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:54.560578 23621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:54.566404 23621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:54.575548 23621 catalog_manager.cc:1347] Generated new cluster ID: bd663744e4f54b43b54f01bab880fbfc
I20260619 13:05:54.575681 23621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:54.587538 23621 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:54.589448 23621 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:54.605952 23621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6329f9c8cd8476189f04ea3878e9abc: Generated new TSK 0
I20260619 13:05:54.607029 23621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:54.618603 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23537
W20260619 13:05:54.619750 23599 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54433
W20260619 13:05:54.620432 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:54.620581 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 23537
2026-06-19T13:05:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:54 dist-test-slave-tpsm krb5kdc[23635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:54 dist-test-slave-tpsm krb5kdc[23635](info): set up 2 sockets
Jun 19 13:05:54 dist-test-slave-tpsm krb5kdc[23635](info): commencing operation
krb5kdc: starting...
W20260619 13:05:55.731904 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:55 dist-test-slave-tpsm krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:55.996371 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44101
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:44101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:56.332307 23648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:56.332602 23648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:56.332695 23648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:56.339944 23648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:56.340058 23648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:56.340106 23648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:56.340169 23648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:56.340215 23648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:56.350492 23648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:44101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:56.351780 23648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:56.353006 23648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:56.364595 23653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:56.364569 23654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:56.364635 23656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:56.365406 23648 server_base.cc:1034] running on GCE node
I20260619 13:05:56.366160 23648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:56.367606 23648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:56.368858 23648 hybrid_clock.cc:648] HybridClock initialized: now 1781874356368819 us; error 43 us; skew 500 ppm
I20260619 13:05:56.369225 23648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:56 dist-test-slave-tpsm krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874356, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:56.376291 23648 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:56.377928 23648 webserver.cc:460] Webserver started at http://127.1.87.126:39401/ using document root <none> and password file <none>
I20260619 13:05:56.378558 23648 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:56.378654 23648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:56.378904 23648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:56.381847 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "4d8bea385f1e4f0db17749639d038e56"
format_stamp: "Formatted at 2026-06-19 13:05:56 on dist-test-slave-tpsm"
server_key: "362bc958be0e2696c9c987f85bae4041"
server_key_iv: "abefbfb63896779402eb873a6e64c41b"
server_key_version: "encryptionkey@0"
I20260619 13:05:56.382524 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "4d8bea385f1e4f0db17749639d038e56"
format_stamp: "Formatted at 2026-06-19 13:05:56 on dist-test-slave-tpsm"
server_key: "362bc958be0e2696c9c987f85bae4041"
server_key_iv: "abefbfb63896779402eb873a6e64c41b"
server_key_version: "encryptionkey@0"
I20260619 13:05:56.387655 23648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:05:56.391228 23663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:56.392585 23648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:05:56.392742 23648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "4d8bea385f1e4f0db17749639d038e56"
format_stamp: "Formatted at 2026-06-19 13:05:56 on dist-test-slave-tpsm"
server_key: "362bc958be0e2696c9c987f85bae4041"
server_key_iv: "abefbfb63896779402eb873a6e64c41b"
server_key_version: "encryptionkey@0"
I20260619 13:05:56.392906 23648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:56.398518 23648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:56.402271 23648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:56.402513 23648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:56.420926 23648 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:44101
I20260619 13:05:56.420972 23714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:44101 every 8 connection(s)
I20260619 13:05:56.422264 23648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:56.425747 23715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:56.431027 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 23648
I20260619 13:05:56.431186 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:56.431710 1373 external_mini_cluster.cc:1428] Setting key 1c01e37294240cbce3e3add271846a6b
I20260619 13:05:56.436718 23715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56: Bootstrap starting.
I20260619 13:05:56.439006 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:56.433403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42917 (local address 127.1.87.126:44101)
0619 13:05:56.434215 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:56.434227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:56.434273 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:05:56.435944 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:56.435964 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:56.436269 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:56.436567 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:56.437591 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:56.437597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:56.437609 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:56.437622 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:56.437678 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:56.437697 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:56.437870 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:56.437960 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 13:05:56.438137 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":398,"threads_started":1}
I20260619 13:05:56.440059 23715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:56.441056 23715 log.cc:824] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:56.442580 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:56.444371 23715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56: No bootstrap required, opened a new log
I20260619 13:05:56.450094 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:56.447534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41749 (local address 127.1.87.126:44101)
0619 13:05:56.447738 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:56.447744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:56.448144 (+ 400us) server_negotiation.cc:407] Connection header received
0619 13:05:56.448271 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:56.448276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:56.448344 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:56.448474 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:56.449434 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:56.449444 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:56.449449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:05:56.449467 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:56.449511 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:05:56.449535 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:56.449662 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:56.449747 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 13:05:56.449802 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:05:56.453325 23715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8bea385f1e4f0db17749639d038e56" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44101 } }
I20260619 13:05:56.453660 23715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:05:56.453760 23715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8bea385f1e4f0db17749639d038e56, State: Initialized, Role: FOLLOWER
I20260619 13:05:56.454282 23715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8bea385f1e4f0db17749639d038e56" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44101 } }
I20260619 13:05:56.454483 23715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:05:56.454578 23715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:05:56.454699 23715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:56.457288 23715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8bea385f1e4f0db17749639d038e56" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44101 } }
I20260619 13:05:56.457768 23715 leader_election.cc:302] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d8bea385f1e4f0db17749639d038e56; no voters:
I20260619 13:05:56.458292 23715 leader_election.cc:288] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:05:56.458448 23725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:56.459129 23725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8bea385f1e4f0db17749639d038e56, State: Running, Role: LEADER
I20260619 13:05:56.459832 23725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8bea385f1e4f0db17749639d038e56" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44101 } }
I20260619 13:05:56.460798 23715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:56.463033 23727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d8bea385f1e4f0db17749639d038e56. Latest consensus state: current_term: 1 leader_uuid: "4d8bea385f1e4f0db17749639d038e56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8bea385f1e4f0db17749639d038e56" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44101 } } }
I20260619 13:05:56.463444 23727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:56.465180 23733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:56.465058 23726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d8bea385f1e4f0db17749639d038e56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8bea385f1e4f0db17749639d038e56" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44101 } } }
I20260619 13:05:56.465363 23726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56 [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:56.470752 23733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:56.481038 23733 catalog_manager.cc:1347] Generated new cluster ID: 991e09ecba5344028cb65b55bd4889dc
I20260619 13:05:56.481186 23733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:56.494619 23733 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:56.495045 23733 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:56.507266 23733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d8bea385f1e4f0db17749639d038e56: Generated new TSK 0
I20260619 13:05:56.508136 23733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:56.521106 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23648
W20260619 13:05:56.522181 23710 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42917
W20260619 13:05:56.522868 1373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23648: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:05:56.523016 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 23648
2026-06-19T13:05:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1898 ms)
[----------] 18 tests from AuthTokenIssuingTest (35071 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:05:56 dist-test-slave-tpsm krb5kdc[23746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:05:56 dist-test-slave-tpsm krb5kdc[23746](info): set up 2 sockets
Jun 19 13:05:56 dist-test-slave-tpsm krb5kdc[23746](info): commencing operation
krb5kdc: starting...
W20260619 13:05:57.632850 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:05:57 dist-test-slave-tpsm krb5kdc[23746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:05:57.893102 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:46005
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:46005,127.1.87.125:33099,127.1.87.124:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:05:58.221077 23759 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:05:58.221396 23759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:58.221459 23759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:58.221534 23759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:58.228713 23759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:58.228798 23759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:58.228855 23759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:58.228906 23759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:58.228960 23759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:58.238639 23759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:46005,127.1.87.125:33099,127.1.87.124:46139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:46005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:58.239889 23759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:58.241098 23759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:58.252669 23765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:58.252755 23764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:58.252709 23767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:58.253585 23759 server_base.cc:1034] running on GCE node
I20260619 13:05:58.254320 23759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:58.255700 23759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:58.256947 23759 hybrid_clock.cc:648] HybridClock initialized: now 1781874358256895 us; error 28 us; skew 500 ppm
I20260619 13:05:58.257272 23759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:58 dist-test-slave-tpsm krb5kdc[23746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:58.260831 23759 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:05:58.262264 23759 webserver.cc:460] Webserver started at http://127.1.87.126:36817/ using document root <none> and password file <none>
I20260619 13:05:58.262923 23759 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:58.263054 23759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:58.263434 23759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:58.266273 23759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "fe2d5f82dcf2428bae3bcb0da652f83b"
format_stamp: "Formatted at 2026-06-19 13:05:58 on dist-test-slave-tpsm"
server_key: "8c052d7aa6e2b96321f50a92f515636c"
server_key_iv: "81510dea908903110454a9d3b7b33e13"
server_key_version: "encryptionkey@0"
I20260619 13:05:58.267130 23759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "fe2d5f82dcf2428bae3bcb0da652f83b"
format_stamp: "Formatted at 2026-06-19 13:05:58 on dist-test-slave-tpsm"
server_key: "8c052d7aa6e2b96321f50a92f515636c"
server_key_iv: "81510dea908903110454a9d3b7b33e13"
server_key_version: "encryptionkey@0"
I20260619 13:05:58.272146 23759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:05:58.275421 23774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:58.276762 23759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:05:58.276944 23759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "fe2d5f82dcf2428bae3bcb0da652f83b"
format_stamp: "Formatted at 2026-06-19 13:05:58 on dist-test-slave-tpsm"
server_key: "8c052d7aa6e2b96321f50a92f515636c"
server_key_iv: "81510dea908903110454a9d3b7b33e13"
server_key_version: "encryptionkey@0"
I20260619 13:05:58.277172 23759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:58.331661 23759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:58.336262 23759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:58.336558 23759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:58.355686 23759 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:46005
I20260619 13:05:58.355705 23825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:46005 every 8 connection(s)
I20260619 13:05:58.357049 23759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:05:58.358278 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 23759
I20260619 13:05:58.358485 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:05:58.359269 1373 external_mini_cluster.cc:1428] Setting key a62f07508cc893490bdf20b8df3f4946
I20260619 13:05:58.361644 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:58.370992 23826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.87.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.125@KRBTEST.COM" created.
Jun 19 13:05:58 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:58.398098 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.374116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.1.87.126:46005)
0619 13:05:58.375965 (+ 1849us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:58.375971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:58.376010 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:05:58.378229 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:58.378235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:58.378468 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:58.378638 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:58.380413 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.381750 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.384137 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.384704 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.384838 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:58.390780 (+ 5942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:58.390788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:58.390801 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:58.390829 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:58.393699 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:58.394426 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:58.394433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:58.394435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:58.394580 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:58.394962 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:58.394968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:58.394976 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:58.395548 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:58.395766 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:58.396283 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 13:05:58.396612 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1670,"spinlock_wait_cycles":21376,"thread_start_us":1422,"threads_started":1}
I20260619 13:05:58.398144 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.374116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:46005 (local address 127.0.0.1:59508)
0619 13:05:58.375017 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.375072 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:58.377840 (+ 2768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:58.378717 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:58.378727 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:58.379647 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:58.380163 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:58.380189 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.381900 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.381908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:58.383969 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:58.383988 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.384977 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.384986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:58.385232 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:58.385957 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:58.385996 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:58.390537 (+ 4541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:58.393849 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:58.393859 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:58.393873 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:58.394282 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:58.394687 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:58.394694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:58.394700 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:58.394850 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:58.395689 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:58.395697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:58.395906 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 13:05:58.396154 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 13:05:58.396733 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":146,"thread_start_us":374,"threads_started":1}
I20260619 13:05:58.401268 23826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:05:58.402424 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.402053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34064)
0619 13:05:58.402210 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.402311 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 13:05:58.402701 23776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
I20260619 13:05:58.404277 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.403914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34076)
0619 13:05:58.404065 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.404141 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:05:58.404459 23826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:33099: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab.
Entry for principal kudu/127.1.87.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab.
I20260619 13:05:58.419859 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.125:33099
--webserver_interface=127.1.87.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab
--principal=kudu/127.1.87.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:46005,127.1.87.125:33099,127.1.87.124:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:05:58.453766 23826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } attempt: 1
I20260619 13:05:58.454892 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.454453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34088)
0619 13:05:58.454636 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.454761 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 13:05:58.455933 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.455586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34096)
0619 13:05:58.455725 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.455794 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 13:05:58.456137 23826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:33099: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
I20260619 13:05:58.524427 23826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } attempt: 2
I20260619 13:05:58.525676 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.525204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34106)
0619 13:05:58.525428 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.525517 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 13:05:58.526857 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.526516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34110)
0619 13:05:58.526667 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.526743 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 13:05:58.527185 23826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:33099: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
I20260619 13:05:58.618494 23826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } attempt: 3
I20260619 13:05:58.619733 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.619286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34118)
0619 13:05:58.619513 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.619602 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 13:05:58.620739 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.620439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34132)
0619 13:05:58.620568 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.620629 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 13:05:58.620921 23826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:33099: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
W20260619 13:05:58.746995 23833 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:05:58.747531 23833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:58.747699 23833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:58.747860 23833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:58.755892 23833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:58.756035 23833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:58.756155 23833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:58.756248 23833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:58.756366 23833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:58.764174 23826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } attempt: 4
I20260619 13:05:58.765791 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.765161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34140)
0619 13:05:58.765538 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.765638 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260619 13:05:58.766939 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.766562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34152)
0619 13:05:58.766722 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.766815 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 13:05:58.767128 23826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:33099: Network error: Client connection negotiation failed: client connection to 127.1.87.125:33099: connect: Connection refused (error 111)
I20260619 13:05:58.769340 23833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:46005,127.1.87.125:33099,127.1.87.124:46139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab
--principal=kudu/127.1.87.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.125:33099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.87.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:58.770709 23833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:58.772181 23833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:05:58.785597 23838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:58.785640 23841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:58.786132 23839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:58.786520 23833 server_base.cc:1034] running on GCE node
I20260619 13:05:58.787446 23833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:58.788841 23833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:58.790091 23833 hybrid_clock.cc:648] HybridClock initialized: now 1781874358790026 us; error 39 us; skew 500 ppm
I20260619 13:05:58.790505 23833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:58 dist-test-slave-tpsm krb5kdc[23746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:58.793648 23833 init.cc:377] Logged in from keytab as kudu/127.1.87.125@KRBTEST.COM (short username kudu)
I20260619 13:05:58.795143 23833 webserver.cc:460] Webserver started at http://127.1.87.125:42953/ using document root <none> and password file <none>
I20260619 13:05:58.795919 23833 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:58.796053 23833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:58.796384 23833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:58.799470 23833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/data/instance:
uuid: "d8e31ae301dc4e0988d336e05f9f5e63"
format_stamp: "Formatted at 2026-06-19 13:05:58 on dist-test-slave-tpsm"
server_key: "a92619a3c064a572712d7e1f35b716ad"
server_key_iv: "a0fca13e9ab89d3a26fee663461f4e09"
server_key_version: "encryptionkey@0"
I20260619 13:05:58.800282 23833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/wal/instance:
uuid: "d8e31ae301dc4e0988d336e05f9f5e63"
format_stamp: "Formatted at 2026-06-19 13:05:58 on dist-test-slave-tpsm"
server_key: "a92619a3c064a572712d7e1f35b716ad"
server_key_iv: "a0fca13e9ab89d3a26fee663461f4e09"
server_key_version: "encryptionkey@0"
I20260619 13:05:58.805682 23833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:05:58.809342 23848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:58.810767 23833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:05:58.810973 23833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/wal
uuid: "d8e31ae301dc4e0988d336e05f9f5e63"
format_stamp: "Formatted at 2026-06-19 13:05:58 on dist-test-slave-tpsm"
server_key: "a92619a3c064a572712d7e1f35b716ad"
server_key_iv: "a0fca13e9ab89d3a26fee663461f4e09"
server_key_version: "encryptionkey@0"
I20260619 13:05:58.811206 23833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:58.829499 23833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:58.833575 23833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:58.833868 23833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:58.853153 23833 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.125:33099
I20260619 13:05:58.853173 23899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.125:33099 every 8 connection(s)
I20260619 13:05:58.854532 23833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
I20260619 13:05:58.854938 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 23833
I20260619 13:05:58.855139 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-1/wal/instance
I20260619 13:05:58.855931 1373 external_mini_cluster.cc:1428] Setting key 830c3389ea4e8f585b0754351f9d3c87
I20260619 13:05:58.859453 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:58.868935 23900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.87.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.124@KRBTEST.COM" created.
Jun 19 13:05:58 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:58.895956 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.871782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.1.87.126:46005)
0619 13:05:58.871993 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:58.871999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:58.873286 (+ 1287us) server_negotiation.cc:407] Connection header received
0619 13:05:58.876253 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:58.876258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:58.876331 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:58.876422 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:58.878547 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.879438 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.881990 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.882468 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.882578 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:58.888342 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:58.888352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:58.888360 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:58.888411 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:58.892230 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:58.893247 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:58.893255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:58.893261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:58.893416 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:58.893880 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:58.893888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:58.893894 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:58.894351 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:58.894511 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:58.895216 (+ 705us) server_negotiation.cc:299] Negotiation successful
0619 13:05:58.895527 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:05:58.896701 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.872252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:46005 (local address 127.0.0.1:59512)
0619 13:05:58.873090 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.873153 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:58.875926 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:58.876703 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:58.876716 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:58.877522 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:58.878342 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:58.878370 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.879636 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.879644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:58.881837 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:58.881854 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.882730 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.882738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:58.883064 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:58.883852 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:58.883891 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:58.888063 (+ 4172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:58.892504 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:58.892511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:58.892527 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:58.893100 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:58.893539 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:58.893543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:58.893545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:58.893748 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:58.894477 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:58.894483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:58.894837 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 13:05:58.895066 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 13:05:58.895545 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":322,"threads_started":1}
I20260619 13:05:58.898552 23900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:05:58 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
I20260619 13:05:58.916199 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.899475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34156)
0619 13:05:58.899701 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.899727 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:58.899835 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:58.900843 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:58.900850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:58.901351 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:58.901557 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:58.901570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.903024 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.903155 (+ 131us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:58.904950 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:58.904969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.905879 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.905887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:58.906106 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:58.906821 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:58.906848 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:58.909815 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:58.913447 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:58.913454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:58.913459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:58.913961 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:58.914398 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:58.914409 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:58.914413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:58.914543 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:58.915202 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:58.915206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:58.915419 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 13:05:58.915572 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 13:05:58.915797 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 13:05:58.916493 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.899657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.1.87.125:33099)
0619 13:05:58.900340 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:58.900346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:58.900445 (+ 99us) server_negotiation.cc:407] Connection header received
0619 13:05:58.900495 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:58.900501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:58.900676 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:58.900791 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:58.901750 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.902875 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.905121 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:58.905706 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:58.905813 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:58.910072 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:58.910087 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:58.910103 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:58.910138 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:58.913292 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:58.914107 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:58.914116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:58.914118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:58.914233 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:58.914687 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:58.914691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:58.914693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:58.915080 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:58.915248 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:58.915736 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 13:05:58.915979 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":389,"threads_started":1}
I20260619 13:05:58.918764 23900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:05:58.919932 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.919432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57514)
0619 13:05:58.919593 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.919726 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 13:05:58.920225 23850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab.
Entry for principal kudu/127.1.87.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab.
I20260619 13:05:58.921979 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.921537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57528)
0619 13:05:58.921720 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.921819 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 13:05:58.922240 23900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:46139: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
I20260619 13:05:58.922498 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.124:46139
--webserver_interface=127.1.87.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab
--principal=kudu/127.1.87.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:43341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:46005,127.1.87.125:33099,127.1.87.124:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:05:58.971611 23900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } attempt: 1
I20260619 13:05:58.972890 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.972427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57534)
0619 13:05:58.972634 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.972747 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 13:05:58.973941 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:58.973595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57538)
0619 13:05:58.973745 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:58.973840 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:05:58.974162 23900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:46139: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
I20260619 13:05:59.042379 23900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } attempt: 2
I20260619 13:05:59.043567 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.043049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57544)
0619 13:05:59.043265 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.043411 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 13:05:59.044538 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.044213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57552)
0619 13:05:59.044364 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.044433 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:05:59.044741 23900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:46139: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
I20260619 13:05:59.066391 23826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } attempt: 5
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
I20260619 13:05:59.081408 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.067169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34160)
0619 13:05:59.067434 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.067458 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:59.067573 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:59.067916 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:59.067919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:59.068317 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:59.068497 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.068506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.069924 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.069929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.070823 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.070836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.071901 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.071906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.072068 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.072764 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:59.072782 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:59.075012 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:59.078823 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.078828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.078831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.079210 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.079699 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.079702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.079705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.079833 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.080592 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:59.080595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:59.080748 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 13:05:59.080898 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 13:05:59.081159 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 13:05:59.081594 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.067362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.1.87.125:33099)
0619 13:05:59.067569 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.067575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.067600 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:05:59.067675 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.067681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.067782 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.067892 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.068660 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.069755 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.070976 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.071739 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.071876 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.075211 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.075218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.075223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.075262 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.078677 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.079364 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.079373 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.079378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.079546 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.079969 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.079977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.079981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.080453 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.080628 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.081046 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.081273 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:05:59.082778 23826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:05:59.083837 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.083451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57558)
0619 13:05:59.083610 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.083683 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 13:05:59.084836 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.084506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57568)
0619 13:05:59.084662 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.084737 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:05:59.085112 23826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:46139: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
I20260619 13:05:59.135969 23900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } attempt: 3
I20260619 13:05:59.136339 23826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } attempt: 1
I20260619 13:05:59.137344 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.136789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57582)
0619 13:05:59.137035 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.137151 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 13:05:59.137450 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.136940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57584)
0619 13:05:59.137183 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.137278 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260619 13:05:59.138401 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.138055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57588)
0619 13:05:59.138187 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.138275 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 13:05:59.138583 23826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:46139: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
I20260619 13:05:59.138914 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.138349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57604)
0619 13:05:59.138688 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.138779 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
W20260619 13:05:59.139139 23900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:46139: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
I20260619 13:05:59.206828 23826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } attempt: 2
I20260619 13:05:59.208137 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.207513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57618)
0619 13:05:59.207791 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.207926 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 13:05:59.209184 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.208814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57630)
0619 13:05:59.208963 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.209055 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:05:59.209393 23826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:46139: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
W20260619 13:05:59.257444 23907 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:05:59.257774 23907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:05:59.257841 23907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:05:59.257923 23907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:05:59.265600 23907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:05:59.265720 23907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:05:59.265781 23907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:05:59.265833 23907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:05:59.265894 23907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:05:59.276042 23907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:43341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:46005,127.1.87.125:33099,127.1.87.124:46139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab
--principal=kudu/127.1.87.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.124:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.87.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:05:59.277359 23907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:05:59.278617 23907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:05:59.282347 23900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } attempt: 4
I20260619 13:05:59.283542 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.282954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57646)
0619 13:05:59.283203 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.283309 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 13:05:59.284696 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.284262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57660)
0619 13:05:59.284429 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.284544 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:05:59.284974 23900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:46139: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
W20260619 13:05:59.291225 23912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:59.291252 23913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:05:59.291507 23915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:05:59.291930 23907 server_base.cc:1034] running on GCE node
I20260619 13:05:59.292856 23907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:05:59.294328 23907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:05:59.295596 23907 hybrid_clock.cc:648] HybridClock initialized: now 1781874359295528 us; error 39 us; skew 500 ppm
I20260619 13:05:59.295984 23907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:05:59.299234 23907 init.cc:377] Logged in from keytab as kudu/127.1.87.124@KRBTEST.COM (short username kudu)
I20260619 13:05:59.300810 23907 webserver.cc:460] Webserver started at http://127.1.87.124:43111/ using document root <none> and password file <none>
I20260619 13:05:59.301514 23907 fs_manager.cc:359] Metadata directory not provided
I20260619 13:05:59.301643 23907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:05:59.301966 23907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:05:59.305294 23907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/data/instance:
uuid: "53d0e68d6b91422dac9e07c88755dbaf"
format_stamp: "Formatted at 2026-06-19 13:05:59 on dist-test-slave-tpsm"
server_key: "c97e75b0b230e4ed4cd4483e40b9c310"
server_key_iv: "d5731cc350d4fdb9fab6794df923998c"
server_key_version: "encryptionkey@0"
I20260619 13:05:59.306114 23907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/wal/instance:
uuid: "53d0e68d6b91422dac9e07c88755dbaf"
format_stamp: "Formatted at 2026-06-19 13:05:59 on dist-test-slave-tpsm"
server_key: "c97e75b0b230e4ed4cd4483e40b9c310"
server_key_iv: "d5731cc350d4fdb9fab6794df923998c"
server_key_version: "encryptionkey@0"
I20260619 13:05:59.311738 23907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:05:59.315254 23922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:05:59.315601 23826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } attempt: 3
I20260619 13:05:59.316814 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.316336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57672)
0619 13:05:59.316564 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.316664 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 13:05:59.316990 23907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:05:59.317166 23907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/wal
uuid: "53d0e68d6b91422dac9e07c88755dbaf"
format_stamp: "Formatted at 2026-06-19 13:05:59 on dist-test-slave-tpsm"
server_key: "c97e75b0b230e4ed4cd4483e40b9c310"
server_key_iv: "d5731cc350d4fdb9fab6794df923998c"
server_key_version: "encryptionkey@0"
I20260619 13:05:59.317425 23907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:05:59.317909 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.317540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57682)
0619 13:05:59.317715 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.317811 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:05:59.318149 23826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:46139: Network error: Client connection negotiation failed: client connection to 127.1.87.124:46139: connect: Connection refused (error 111)
I20260619 13:05:59.336921 23907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:05:59.341327 23907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:05:59.341629 23907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:05:59.360410 23907 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.124:46139
I20260619 13:05:59.360426 23973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.124:46139 every 8 connection(s)
I20260619 13:05:59.361720 23907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
I20260619 13:05:59.365567 23974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:05:59.367380 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 23907
I20260619 13:05:59.367558 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874004469744-1373-0/minicluster-data/master-2/wal/instance
I20260619 13:05:59.367995 1373 external_mini_cluster.cc:1428] Setting key e3545f9a981acec766fe62146a93e93a
I20260619 13:05:59.372951 23974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:59.392896 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.369751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.1.87.124:46139)
0619 13:05:59.370576 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.370587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.370631 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:05:59.372327 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.372414 (+ 87us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.372824 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.373135 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.374984 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.376843 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.378565 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.379536 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.379675 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.384943 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.384952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.384964 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.384999 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.388157 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.389112 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.389135 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.389142 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.389302 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.389761 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.389764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.389766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.390288 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.390549 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.391071 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.391559 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":412,"threads_started":1}
I20260619 13:05:59.393520 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.376164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:46005 (local address 127.0.0.1:59518)
0619 13:05:59.376871 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.376908 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:59.377163 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:59.377652 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:59.377659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:59.378451 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:59.378862 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.378872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.380121 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.380129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.381582 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.381597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.382839 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.382848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.383090 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.383902 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:59.383938 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:59.386907 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:59.390572 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.390581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.390588 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.391137 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.391625 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.391629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.391631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.391787 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.392390 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:59.392395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:59.392566 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 13:05:59.392727 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 13:05:59.393001 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":355,"threads_started":1}
I20260619 13:05:59.393523 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.376137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.1.87.126:46005)
0619 13:05:59.376318 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.376323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.377022 (+ 699us) server_negotiation.cc:407] Connection header received
0619 13:05:59.377322 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.377326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.377477 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.377564 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.379009 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.379813 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.381721 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.382549 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.382708 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.387152 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.387158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.387161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.387198 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.390433 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.391284 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.391291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.391295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.391491 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.391903 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.391908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.391911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.392239 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.392396 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.392854 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.393091 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:05:59.395081 23974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:05:59.409543 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.396017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:33099 (local address 127.0.0.1:34176)
0619 13:05:59.396355 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.396372 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:59.396482 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:59.396807 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:59.396812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:59.397164 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:59.397308 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.397316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.398725 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.398730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.399850 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.399862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.400773 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.400779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.400957 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.401575 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:59.401591 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:59.403603 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:59.406986 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.406993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.406997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.407561 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.407928 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.407933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.407939 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.408122 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.408594 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:59.408599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:59.408781 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 13:05:59.408942 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 13:05:59.409246 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260619 13:05:59.409821 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.395987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.1.87.125:33099)
0619 13:05:59.396225 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.396231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.396460 (+ 229us) server_negotiation.cc:407] Connection header received
0619 13:05:59.396594 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.396600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.396678 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.396790 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.397465 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.398586 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.399986 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.400635 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.400730 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.403807 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.403817 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.403823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.403867 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.406848 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.407678 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.407682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.407684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.407815 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.408237 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.408240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.408242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.408489 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.408603 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.409036 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.409259 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 13:05:59.410751 23974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:05:59.411271 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.396393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.1.87.126:46005)
0619 13:05:59.396559 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.396565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.396582 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:05:59.396630 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.396633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.396682 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.396784 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.398240 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.399141 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.400509 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.401052 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.401193 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.405194 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.405199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.405203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.405242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.408365 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.409183 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.409190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.409194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.409362 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.409788 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.409793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.409802 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.410145 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.410441 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.410714 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.410924 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
I20260619 13:05:59.424968 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.411651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.1.87.124:46139)
0619 13:05:59.411805 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.411809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.411824 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:05:59.411985 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.411988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.412040 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.412113 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.412958 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.413892 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.415266 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.415826 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.415969 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.419468 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.419476 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.419482 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.419523 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.422544 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.423150 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.423155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.423159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.423265 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.423634 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.423641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.423644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.423912 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.424051 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.424392 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.424598 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 13:05:59.425163 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.411479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57706)
0619 13:05:59.411758 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.411780 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:59.411884 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:59.412175 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:59.412186 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:59.412639 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:59.412834 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.412845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.414006 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.414014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.415130 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.415144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.415994 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.416000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.416211 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.416757 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:59.416776 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:59.419223 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:59.422688 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.422693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.422697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.423031 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.423404 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.423409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.423413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.423535 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.424029 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:59.424033 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:59.424171 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 13:05:59.424310 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 13:05:59.424746 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":134}
I20260619 13:05:59.428218 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.413003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.1.87.125:33099)
0619 13:05:59.413173 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.413179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.413197 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:05:59.413308 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.413312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.413383 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.413477 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.414583 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.415579 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.417073 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.417691 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.417839 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.422639 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.422650 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.422654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.422684 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.425538 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.426318 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.426324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.426327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.426463 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.426839 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.426844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.426847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.427112 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.427426 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.427677 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.427899 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:05:59.430514 23974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf: Bootstrap starting.
I20260619 13:05:59.433002 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:05:59.434245 23974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:59.435489 23974 log.cc:824] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:59.438743 23974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf: No bootstrap required, opened a new log
I20260619 13:05:59.435626 23801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534
I20260619 13:05:59.449776 23974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } }
I20260619 13:05:59.450213 23974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d0e68d6b91422dac9e07c88755dbaf, State: Initialized, Role: FOLLOWER
I20260619 13:05:59.450704 23974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } }
I20260619 13:05:59.451676 23980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.452018 23980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:05:59.452973 23974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:05:59.463052 23991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:05:59.463198 23991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:05:59.460595 23907 master_runner.cc:186] Error getting master registration for 127.1.87.126:46005: OK, instance_id { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" instance_seqno: 1781874358350303 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:05:59.473078 23907 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:05:59.495381 23826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } attempt: 4
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
I20260619 13:05:59.509274 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.496225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.1.87.124:46139)
0619 13:05:59.496524 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.496531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.496556 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:05:59.496625 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.496630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.496710 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.496839 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.497570 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.498332 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.499664 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.500108 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.500246 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.503753 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.503758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.503761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.503796 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.506571 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.507218 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.507226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.507232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.507409 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.507777 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.507784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.507787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.508139 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.508303 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.508687 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.508902 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:05:59.509272 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.496185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57708)
0619 13:05:59.496376 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.496396 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:59.496510 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:59.496848 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:59.496851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:59.497241 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:59.497400 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.497409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.498485 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.498491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.499515 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.499528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.500269 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.500275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.500487 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.501120 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:59.501137 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:59.503537 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:59.506713 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.506721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.506726 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.507100 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.507526 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.507532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.507536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.507674 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.508280 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:59.508286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:59.508434 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 13:05:59.508571 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 13:05:59.508872 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 13:05:59.515928 23826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b: Bootstrap starting.
I20260619 13:05:59.520205 23826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:59.521692 23826 log.cc:824] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:59.525782 23826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b: No bootstrap required, opened a new log
I20260619 13:05:59.533852 23826 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } }
I20260619 13:05:59.534320 23826 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2d5f82dcf2428bae3bcb0da652f83b, State: Initialized, Role: FOLLOWER
I20260619 13:05:59.534823 23826 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } }
I20260619 13:05:59.535897 23994 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.536288 23994 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:05:59.537179 23826 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:59.537220 23801 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 13:05:59.537416 23801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:59.539983 23801 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } }
I20260619 13:05:59.541237 23801 leader_election.cc:288] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [CANDIDATE]: Term 1 election: Requested vote from peers d8e31ae301dc4e0988d336e05f9f5e63 (127.1.87.125:33099), 53d0e68d6b91422dac9e07c88755dbaf (127.1.87.124:46139)
I20260619 13:05:59.542887 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "53d0e68d6b91422dac9e07c88755dbaf"
I20260619 13:05:59.543442 23949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:59.548965 23949 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe2d5f82dcf2428bae3bcb0da652f83b in term 1.
I20260619 13:05:59.550165 23776 leader_election.cc:302] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53d0e68d6b91422dac9e07c88755dbaf, fe2d5f82dcf2428bae3bcb0da652f83b; no voters:
I20260619 13:05:59.550782 23994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:05:59.563494 23994 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [term 1 LEADER]: Becoming Leader. State: Replica: fe2d5f82dcf2428bae3bcb0da652f83b, State: Running, Role: LEADER
I20260619 13:05:59.565148 23994 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } }
I20260619 13:05:59.581759 23999 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2d5f82dcf2428bae3bcb0da652f83b. Latest consensus state: current_term: 1 leader_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.582302 23999 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:59.584194 23900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } attempt: 5
W20260619 13:05:59.589413 23780 tablet.cc:2369] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:05:59.593850 23928 tablet.cc:2369] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 13:05:59 dist-test-slave-tpsm krb5kdc[23746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
I20260619 13:05:59.610255 24016 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 13:05:59.616480 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.576218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.1.87.126:46005)
0619 13:05:59.576461 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.576471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.588089 (+ 11618us) server_negotiation.cc:407] Connection header received
0619 13:05:59.588406 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.588415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.588499 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.588579 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.591515 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.592982 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.595233 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.596314 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.596439 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.606765 (+ 10326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.606776 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.606784 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.606844 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.611473 (+ 4629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.613077 (+ 1604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.613082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.613086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.613294 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.614479 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.614484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.614487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.614882 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.615132 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.615789 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.616064 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 13:05:59.614656 23759 master_runner.cc:186] Error getting master registration for 127.1.87.125:33099: OK, instance_id { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" instance_seqno: 1781874358847598 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:05:59.625228 23759 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:05:59.629069 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.584898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.1.87.124:46139)
0619 13:05:59.585060 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.585064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.585084 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:05:59.585236 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.585244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.585343 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.585450 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.587012 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.588464 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.590833 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.591840 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.592004 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.615425 (+ 23421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.615482 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.615491 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.615554 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.620793 (+ 5239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.623618 (+ 2825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.623627 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.623633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.623846 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.626108 (+ 2262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.626139 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.626143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.626769 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.627095 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.627962 (+ 867us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.628413 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:05:59.630344 24011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:05:59.630419 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.584784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:46139 (local address 127.0.0.1:57714)
0619 13:05:59.584933 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 13:05:59.584952 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:05:59.585072 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:05:59.585713 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:05:59.585720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:05:59.586385 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:05:59.586651 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.586667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.588723 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.588732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.590566 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:05:59.590590 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.592364 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.592373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:05:59.592897 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.593858 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:05:59.593907 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:05:59.612691 (+ 18784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:05:59.621173 (+ 8482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.621182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.621199 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.623419 (+ 2220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.624864 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:05:59.624873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:05:59.624879 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:05:59.625926 (+ 1047us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:05:59.627226 (+ 1300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:05:59.627232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:05:59.627495 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 13:05:59.629360 (+ 1865us) client_negotiation.cc:240] Negotiation successful
0619 13:05:59.629845 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 13:05:59.636584 23900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63: Bootstrap starting.
I20260619 13:05:59.641211 23900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63: Neither blocks nor log segments found. Creating new log.
I20260619 13:05:59.642640 23900 log.cc:824] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63: Log is configured to *not* fsync() on all Append() calls
I20260619 13:05:59.657627 24011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:05:59.671290 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.582965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.1.87.125:33099)
0619 13:05:59.583282 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.583291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.594667 (+ 11376us) server_negotiation.cc:407] Connection header received
0619 13:05:59.594871 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.594879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.594978 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.595070 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.598483 (+ 3413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.599598 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.651815 (+ 52217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.652918 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.653036 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.662395 (+ 9359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.662402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.662406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.662450 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.666426 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.668121 (+ 1695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.668131 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.668138 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.668384 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.669072 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.669080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.669084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.669533 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.669810 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.670653 (+ 843us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.670920 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260619 13:05:59.673254 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:05:59.594223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.1.87.124:46139)
0619 13:05:59.628540 (+ 34317us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.628550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.628581 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:05:59.628652 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.628658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.628773 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.636100 (+ 7327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.636486 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.639609 (+ 3123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.643788 (+ 4179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:05:59.644744 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:05:59.651490 (+ 6746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:05:59.659538 (+ 8048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:05:59.659545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:05:59.659551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:05:59.659598 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:05:59.665779 (+ 6181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.669114 (+ 3335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.669122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.669127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.669346 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:05:59.669810 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:05:59.669815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:05:59.669819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:05:59.670199 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:05:59.672409 (+ 2210us) server_negotiation.cc:1035] Waiting for connection context
0619 13:05:59.672557 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 13:05:59.672794 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34105,"thread_start_us":35485,"threads_started":1}
I20260619 13:05:59.678839 23900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63: No bootstrap required, opened a new log
I20260619 13:05:59.691073 23900 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } }
I20260619 13:05:59.691634 23900 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e31ae301dc4e0988d336e05f9f5e63, State: Initialized, Role: FOLLOWER
I20260619 13:05:59.692214 23900 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } }
I20260619 13:05:59.693724 23949 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [term 1 FOLLOWER]: Refusing update from remote peer fe2d5f82dcf2428bae3bcb0da652f83b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:59.694813 23994 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:59.704660 24020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.711476 24020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:05:59.706087 23900 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:05:59.706152 23875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d8e31ae301dc4e0988d336e05f9f5e63"
I20260619 13:05:59.706498 23874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:05:59.717284 23874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [term 1 FOLLOWER]: Refusing update from remote peer fe2d5f82dcf2428bae3bcb0da652f83b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:05:59.722062 23875 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe2d5f82dcf2428bae3bcb0da652f83b in term 1.
I20260619 13:05:59.723286 23994 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:05:59.751219 24020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2d5f82dcf2428bae3bcb0da652f83b. Latest consensus state: current_term: 1 leader_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.751677 24020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:05:59.757773 24020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.758168 24020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:05:59.769796 23999 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.770216 23999 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:59.775795 23980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2d5f82dcf2428bae3bcb0da652f83b. Latest consensus state: current_term: 1 leader_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.776233 23980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:05:59.780045 24032 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.780622 24032 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b [sys.catalog]: This master's current role is: LEADER
I20260619 13:05:59.786279 23980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe2d5f82dcf2428bae3bcb0da652f83b" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 46005 } } peers { permanent_uuid: "d8e31ae301dc4e0988d336e05f9f5e63" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 33099 } } peers { permanent_uuid: "53d0e68d6b91422dac9e07c88755dbaf" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 46139 } } }
I20260619 13:05:59.786585 23980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:05:59.792660 24011 catalog_manager.cc:1347] Generated new cluster ID: 44066e0a629f4d6094bf572606c157fc
I20260619 13:05:59.792775 24011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:05:59.796461 24026 mvcc.cc:205] Tried to move back new op lower bound from 7298557377262125056 to 7298557376796663808. Current Snapshot: MvccSnapshot[applied={T|T < 7298557377262125056 or (T in {7298557377262125056})}]
I20260619 13:05:59.827919 24011 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:05:59.829716 24011 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:05:59.846495 24011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe2d5f82dcf2428bae3bcb0da652f83b: Generated new TSK 0
I20260619 13:05:59.847579 24011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:05:59.857697 24039 catalog_manager.cc:1259] Loaded cluster ID: 44066e0a629f4d6094bf572606c157fc
I20260619 13:05:59.857807 24039 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63: loading cluster ID for follower catalog manager: success
I20260619 13:05:59.861340 24039 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63: acquiring CA information for follower catalog manager: success
I20260619 13:05:59.863677 24039 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8e31ae301dc4e0988d336e05f9f5e63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 13:05:59.989322 23905 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:05:59.966634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.1.87.125:33099)
0619 13:05:59.966846 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.966854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.987259 (+ 20405us) server_negotiation.cc:407] Connection header received
0619 13:05:59.987745 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.987751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.987858 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.987942 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.989058 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34194: BlockingRecv error: recv got EOF from 127.0.0.1:34194 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 13:05:59.990909 23830 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:05:59.960106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.1.87.126:46005)
0619 13:05:59.960360 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.960367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.982599 (+ 22232us) server_negotiation.cc:407] Connection header received
0619 13:05:59.982841 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.982848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.982940 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.983052 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.990676 (+ 7624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59550: BlockingRecv error: recv got EOF from 127.0.0.1:59550 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 13:05:59.992408 24014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:05:59.983373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.1.87.124:46139)
0619 13:05:59.983536 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:05:59.983542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:05:59.990168 (+ 6626us) server_negotiation.cc:407] Connection header received
0619 13:05:59.990864 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:05:59.990870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:05:59.990955 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:05:59.991027 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:05:59.992181 (+ 1154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57730: BlockingRecv error: recv got EOF from 127.0.0.1:57730 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:06:00.037189 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.019530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.1.87.125:33099)
0619 13:06:00.019727 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.019733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.029134 (+ 9401us) server_negotiation.cc:407] Connection header received
0619 13:06:00.029751 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.029759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.029979 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.030131 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.030846 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.032026 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.034044 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.034720 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.034818 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.035618 (+ 800us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.036348 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.036470 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.036713 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.036820 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 13:06:00.083777 24014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:00.070492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.1.87.124:46139)
0619 13:06:00.070700 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.070706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.081819 (+ 11113us) server_negotiation.cc:407] Connection header received
0619 13:06:00.082161 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.082168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.082306 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.082385 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:00.083507 (+ 1122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57744: BlockingRecv error: recv got EOF from 127.0.0.1:57744 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:06:00.137512 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.119796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.1.87.125:33099)
0619 13:06:00.119978 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.119984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.130714 (+ 10730us) server_negotiation.cc:407] Connection header received
0619 13:06:00.131207 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.131215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.131344 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.131413 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.132099 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.133253 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.135256 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.135954 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.136049 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.136652 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.136820 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.136882 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.137107 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.137200 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 13:06:00.147722 24014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:00.145344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.1.87.124:46139)
0619 13:06:00.145522 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.145528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.146010 (+ 482us) server_negotiation.cc:407] Connection header received
0619 13:06:00.146288 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.146294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.146382 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.146456 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:00.147467 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57758: BlockingRecv error: recv got EOF from 127.0.0.1:57758 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:06:00.162518 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.155102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.1.87.125:33099)
0619 13:06:00.155287 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.155293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.155876 (+ 583us) server_negotiation.cc:407] Connection header received
0619 13:06:00.156141 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.156147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.156231 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.156296 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.157017 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.158199 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.160261 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.160906 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.161023 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.161628 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.161786 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.161847 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.162071 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.162153 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 13:06:00.172230 24014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:00.169853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.1.87.124:46139)
0619 13:06:00.170042 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.170048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.170540 (+ 492us) server_negotiation.cc:407] Connection header received
0619 13:06:00.170814 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.170819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.170924 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.170996 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:00.171989 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57766: BlockingRecv error: recv got EOF from 127.0.0.1:57766 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:06:00.187201 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.180155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.1.87.125:33099)
0619 13:06:00.180390 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.180396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.180956 (+ 560us) server_negotiation.cc:407] Connection header received
0619 13:06:00.181155 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.181161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.181254 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.181371 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.181822 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.183049 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.185005 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.185719 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.185864 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.186424 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.186562 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.186682 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.186839 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.186924 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260619 13:06:00.197479 24014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:00.194099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.1.87.124:46139)
0619 13:06:00.194359 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.194365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.195860 (+ 1495us) server_negotiation.cc:407] Connection header received
0619 13:06:00.196055 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.196061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.196172 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.196282 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:00.197242 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57782: BlockingRecv error: recv got EOF from 127.0.0.1:57782 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 13:06:00.222975 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.216368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.1.87.125:33099)
0619 13:06:00.216583 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.216590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.217108 (+ 518us) server_negotiation.cc:407] Connection header received
0619 13:06:00.217328 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.217334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.217427 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.217536 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.218013 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.219125 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.220992 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.221665 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.221788 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.222188 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.222312 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.222392 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.222570 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.222653 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 13:06:00.235690 24014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:00.233413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.1.87.124:46139)
0619 13:06:00.233602 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.233608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.234132 (+ 524us) server_negotiation.cc:407] Connection header received
0619 13:06:00.234292 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.234297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.234397 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.234509 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:00.235433 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57792: BlockingRecv error: recv got EOF from 127.0.0.1:57792 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:06:00.263204 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.256486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.1.87.125:33099)
0619 13:06:00.256687 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.256709 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.257225 (+ 516us) server_negotiation.cc:407] Connection header received
0619 13:06:00.257430 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.257437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.257534 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.257640 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.258095 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.259175 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.261121 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.261805 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.261924 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.262432 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.262581 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.262673 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.262834 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.262923 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 13:06:00.271795 24014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:00.269647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.1.87.124:46139)
0619 13:06:00.269855 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.269861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.270346 (+ 485us) server_negotiation.cc:407] Connection header received
0619 13:06:00.270522 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.270526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.270597 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.270680 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:00.271553 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57794: BlockingRecv error: recv got EOF from 127.0.0.1:57794 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:06:00.313506 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.307436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.1.87.125:33099)
0619 13:06:00.307695 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.307704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.308272 (+ 568us) server_negotiation.cc:407] Connection header received
0619 13:06:00.308407 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.308413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.308505 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.308627 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.308979 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.310160 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.311499 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.312179 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.312305 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.312648 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.312795 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.312890 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.313091 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.313174 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 13:06:00.321630 24014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:00.319550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.1.87.124:46139)
0619 13:06:00.319760 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.319767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.320181 (+ 414us) server_negotiation.cc:407] Connection header received
0619 13:06:00.320381 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.320388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.320487 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.320608 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:00.321376 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57804: BlockingRecv error: recv got EOF from 127.0.0.1:57804 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:06:00.394251 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.388878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.1.87.125:33099)
0619 13:06:00.389130 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.389139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.389657 (+ 518us) server_negotiation.cc:407] Connection header received
0619 13:06:00.389888 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.389892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.389960 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.390063 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.390456 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.391209 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.392541 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.393002 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.393148 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.393561 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.393665 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.393765 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.393909 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.393987 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 13:06:00.402343 24014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:00.400358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.1.87.124:46139)
0619 13:06:00.400580 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.400587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.400819 (+ 232us) server_negotiation.cc:407] Connection header received
0619 13:06:00.401034 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.401038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.401113 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.401211 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:00.402096 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57820: BlockingRecv error: recv got EOF from 127.0.0.1:57820 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 13:06:00.465116 23991 catalog_manager.cc:1259] Loaded cluster ID: 44066e0a629f4d6094bf572606c157fc
I20260619 13:06:00.465230 23991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf: loading cluster ID for follower catalog manager: success
I20260619 13:06:00.468672 23991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf: acquiring CA information for follower catalog manager: success
I20260619 13:06:00.471262 23991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53d0e68d6b91422dac9e07c88755dbaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:06:00.539362 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.533882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.1.87.125:33099)
0619 13:06:00.534055 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.534060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.534436 (+ 376us) server_negotiation.cc:407] Connection header received
0619 13:06:00.534696 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.534701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.534782 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.534868 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.535258 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.535958 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.537331 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.537747 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.537845 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.538403 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.538560 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.538673 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.538872 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.538968 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:06:00.555001 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.547436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.1.87.124:46139)
0619 13:06:00.547667 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.547673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.547694 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:06:00.547768 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.547773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.547994 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.548146 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.548757 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.549917 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.552232 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.552699 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.552838 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.553576 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.554331 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.554501 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.554555 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.554649 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 13:06:00.571484 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.562011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.1.87.125:33099)
0619 13:06:00.562225 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.562267 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.562575 (+ 308us) server_negotiation.cc:407] Connection header received
0619 13:06:00.563094 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.563103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.563232 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.563332 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.564266 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.565696 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.568485 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.569418 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.569539 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.570425 (+ 886us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.570595 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.570682 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.570942 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.571057 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 13:06:00.580911 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.562617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.1.87.124:46139)
0619 13:06:00.562790 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.562799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.574903 (+ 12104us) server_negotiation.cc:407] Connection header received
0619 13:06:00.575233 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.575242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.575435 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.575520 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.576466 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.577174 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.579054 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.579562 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.579675 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.580149 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.580257 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.580347 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.580496 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.580607 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 13:06:00.583166 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:00.561927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.1.87.126:46005)
0619 13:06:00.577475 (+ 15548us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:00.577483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:00.577502 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:06:00.577619 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:00.577626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:00.577842 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:00.578026 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:00.578404 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.579402 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.581237 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:00.581693 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:00.581809 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:00.582291 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:00.582597 (+ 306us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:00.582711 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:00.582814 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 13:06:00.582893 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15401,"thread_start_us":16221,"threads_started":1}
I20260619 13:06:00.588079 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23759
I20260619 13:06:00.589025 23821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:00.699772 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 23759
W20260619 13:06:00.720117 23850 connection.cc:535] server connection from 127.0.0.1:34160 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 13:06:00.720528 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23833
I20260619 13:06:00.721678 23895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:00.827809 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 23833
I20260619 13:06:00.847483 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 23907
I20260619 13:06:00.848522 23969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:00.952446 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 23907
2026-06-19T13:06:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4436 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:06:01 dist-test-slave-tpsm krb5kdc[24153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:06:01 dist-test-slave-tpsm krb5kdc[24153](info): set up 2 sockets
Jun 19 13:06:01 dist-test-slave-tpsm krb5kdc[24153](info): commencing operation
krb5kdc: starting...
W20260619 13:06:02.056011 1373 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/test-user.keytab.
Jun 19 13:06:02 dist-test-slave-tpsm krb5kdc[24153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:06:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:06:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.87.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
Entry for principal kudu/127.1.87.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab.
I20260619 13:06:02.300633 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44483
--webserver_interface=127.1.87.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:44483,127.1.87.125:34407,127.1.87.124:38791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:06:02.624358 24166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:06:02.624732 24166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:02.624857 24166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:02.624992 24166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:02.632067 24166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:06:02.632200 24166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:02.632309 24166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:02.632397 24166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:06:02.632493 24166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:06:02.642086 24166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:44483,127.1.87.125:34407,127.1.87.124:38791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.126.keytab
--principal=kudu/127.1.87.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.126:44483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.87.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:02.643383 24166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:02.644630 24166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:02.655928 24171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:02.655874 24174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:02.655836 24172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:02.656889 24166 server_base.cc:1034] running on GCE node
I20260619 13:06:02.657806 24166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:02.659353 24166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:02.660607 24166 hybrid_clock.cc:648] HybridClock initialized: now 1781874362660539 us; error 53 us; skew 500 ppm
I20260619 13:06:02.660983 24166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:02 dist-test-slave-tpsm krb5kdc[24153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:02.666978 24166 init.cc:377] Logged in from keytab as kudu/127.1.87.126@KRBTEST.COM (short username kudu)
I20260619 13:06:02.668668 24166 webserver.cc:460] Webserver started at http://127.1.87.126:45311/ using document root <none> and password file <none>
I20260619 13:06:02.669384 24166 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:02.669521 24166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:02.669838 24166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:02.672829 24166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/data/instance:
uuid: "e5f5d0a55d8c40fcb28035b9acce484d"
format_stamp: "Formatted at 2026-06-19 13:06:02 on dist-test-slave-tpsm"
server_key: "a1ebd868132e19da47b25a8c57b7240e"
server_key_iv: "2e8c1bad7b680caa8a6d7cedd3507999"
server_key_version: "encryptionkey@0"
I20260619 13:06:02.673662 24166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/wal/instance:
uuid: "e5f5d0a55d8c40fcb28035b9acce484d"
format_stamp: "Formatted at 2026-06-19 13:06:02 on dist-test-slave-tpsm"
server_key: "a1ebd868132e19da47b25a8c57b7240e"
server_key_iv: "2e8c1bad7b680caa8a6d7cedd3507999"
server_key_version: "encryptionkey@0"
I20260619 13:06:02.678793 24166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:06:02.682176 24181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:02.683573 24166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:06:02.683749 24166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/wal
uuid: "e5f5d0a55d8c40fcb28035b9acce484d"
format_stamp: "Formatted at 2026-06-19 13:06:02 on dist-test-slave-tpsm"
server_key: "a1ebd868132e19da47b25a8c57b7240e"
server_key_iv: "2e8c1bad7b680caa8a6d7cedd3507999"
server_key_version: "encryptionkey@0"
I20260619 13:06:02.683962 24166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:02.699671 24166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:02.704032 24166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:02.704301 24166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:02.724257 24166 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.126:44483
I20260619 13:06:02.724241 24232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.126:44483 every 8 connection(s)
I20260619 13:06:02.725591 24166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/data/info.pb
I20260619 13:06:02.729194 24233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:02.734591 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 24166
I20260619 13:06:02.734794 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-0/wal/instance
I20260619 13:06:02.735214 1373 external_mini_cluster.cc:1428] Setting key 8bc1f242390433f06d9870a67d9d0e24
I20260619 13:06:02.735597 24233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:06:02 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.87.125@KRBTEST.COM; defaulting to no policy
I20260619 13:06:02.762652 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.738582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44483 (local address 127.0.0.1:55124)
0619 13:06:02.739502 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:02.739551 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:02.742431 (+ 2880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:02.743392 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:02.743401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:02.744127 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:02.744467 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:02.744483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:02.746894 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:02.746898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:02.748250 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:02.748267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:02.749582 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:02.749587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:02.749766 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:02.750463 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:02.750491 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:02.753637 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:02.757290 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:02.757306 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:02.757323 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:02.758011 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:02.758675 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:02.758681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:02.758684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:02.758926 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:02.759800 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:02.759811 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:02.760101 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 13:06:02.760443 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 13:06:02.761155 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":152,"thread_start_us":324,"threads_started":1}
I20260619 13:06:02.762617 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.739701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.1.87.126:44483)
0619 13:06:02.740303 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:02.740308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:02.740348 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:06:02.742853 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:02.742859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:02.743111 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:02.743275 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:02.744682 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:02.746703 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:02.748397 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:02.749344 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:02.749539 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:02.753939 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:02.753948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:02.753962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:02.754009 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:02.757127 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:02.758193 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:02.758200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:02.758202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:02.758525 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:02.759044 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:02.759048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:02.759050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:02.759648 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:02.759828 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:02.760621 (+ 793us) server_negotiation.cc:299] Negotiation successful
0619 13:06:02.761031 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":439,"thread_start_us":289,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.125@KRBTEST.COM" created.
I20260619 13:06:02.766009 24233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:06:02.767174 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.766785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36352)
0619 13:06:02.766947 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:02.767049 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 13:06:02.767508 24184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
I20260619 13:06:02.769101 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.768749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36360)
0619 13:06:02.768900 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:02.768964 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:06:02.769313 24233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:34407: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab.
Entry for principal kudu/127.1.87.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab.
I20260619 13:06:02.794394 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.125:34407
--webserver_interface=127.1.87.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab
--principal=kudu/127.1.87.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:44483,127.1.87.125:34407,127.1.87.124:38791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:06:02.818662 24233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } attempt: 1
I20260619 13:06:02.820190 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.819575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36366)
0619 13:06:02.819826 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:02.819982 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 13:06:02.821445 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.820926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36372)
0619 13:06:02.821116 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:02.821221 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 13:06:02.821671 24233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:34407: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
I20260619 13:06:02.889953 24233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } attempt: 2
I20260619 13:06:02.891201 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.890683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36388)
0619 13:06:02.890902 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:02.891013 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 13:06:02.892231 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.891834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36394)
0619 13:06:02.891972 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:02.892067 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 13:06:02.892408 24233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:34407: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
I20260619 13:06:02.983693 24233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } attempt: 3
I20260619 13:06:02.985085 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.984483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36402)
0619 13:06:02.984740 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:02.984874 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 13:06:02.986292 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:02.985798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36416)
0619 13:06:02.985974 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:02.986079 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 13:06:02.986517 24233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:34407: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
W20260619 13:06:03.122232 24240 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:06:03.122561 24240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:03.122637 24240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:03.122720 24240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 13:06:03.129784 24233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } attempt: 4
W20260619 13:06:03.130123 24240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:06:03.130221 24240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:03.130321 24240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:03.130388 24240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:06:03.130494 24240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:06:03.131204 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.130601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36424)
0619 13:06:03.130936 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.131040 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260619 13:06:03.132205 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.131817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36428)
0619 13:06:03.131957 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.132049 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 13:06:03.132380 24233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.125:34407: Network error: Client connection negotiation failed: client connection to 127.1.87.125:34407: connect: Connection refused (error 111)
I20260619 13:06:03.141530 24240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:44483,127.1.87.125:34407,127.1.87.124:38791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.125.keytab
--principal=kudu/127.1.87.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.125:34407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.87.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:03.142865 24240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:03.144181 24240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:06:03.155433 24245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:03.155509 24246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:03.155411 24248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:03.161087 24240 server_base.cc:1034] running on GCE node
I20260619 13:06:03.161904 24240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:03.163403 24240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:03.164628 24240 hybrid_clock.cc:648] HybridClock initialized: now 1781874363164561 us; error 37 us; skew 500 ppm
I20260619 13:06:03.164951 24240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:03.168051 24240 init.cc:377] Logged in from keytab as kudu/127.1.87.125@KRBTEST.COM (short username kudu)
I20260619 13:06:03.169601 24240 webserver.cc:460] Webserver started at http://127.1.87.125:36315/ using document root <none> and password file <none>
I20260619 13:06:03.170296 24240 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:03.170383 24240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:03.170737 24240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:03.173689 24240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/data/instance:
uuid: "35dccef21e5248c0a8f2074818e3c554"
format_stamp: "Formatted at 2026-06-19 13:06:03 on dist-test-slave-tpsm"
server_key: "0b08e0aebfcdeaed4d3575b291c272c9"
server_key_iv: "15578cdb1bb63b61b1dfcfb76a869ec7"
server_key_version: "encryptionkey@0"
I20260619 13:06:03.174448 24240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/wal/instance:
uuid: "35dccef21e5248c0a8f2074818e3c554"
format_stamp: "Formatted at 2026-06-19 13:06:03 on dist-test-slave-tpsm"
server_key: "0b08e0aebfcdeaed4d3575b291c272c9"
server_key_iv: "15578cdb1bb63b61b1dfcfb76a869ec7"
server_key_version: "encryptionkey@0"
I20260619 13:06:03.179503 24240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:06:03.182834 24255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:03.184161 24240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:06:03.184340 24240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/wal
uuid: "35dccef21e5248c0a8f2074818e3c554"
format_stamp: "Formatted at 2026-06-19 13:06:03 on dist-test-slave-tpsm"
server_key: "0b08e0aebfcdeaed4d3575b291c272c9"
server_key_iv: "15578cdb1bb63b61b1dfcfb76a869ec7"
server_key_version: "encryptionkey@0"
I20260619 13:06:03.184551 24240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:03.211822 24240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:03.215786 24240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:03.216054 24240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:03.234387 24240 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.125:34407
I20260619 13:06:03.234406 24306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.125:34407 every 8 connection(s)
I20260619 13:06:03.235716 24240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/data/info.pb
I20260619 13:06:03.239166 24307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:03.243088 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 24240
I20260619 13:06:03.243263 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-1/wal/instance
I20260619 13:06:03.243818 1373 external_mini_cluster.cc:1428] Setting key 2122ca8495e7c0c7671f5f98bbe858e3
I20260619 13:06:03.246603 24307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:06:03.268296 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.249690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.1.87.126:44483)
0619 13:06:03.249901 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.249908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.251081 (+ 1173us) server_negotiation.cc:407] Connection header received
0619 13:06:03.252991 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.253000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.253101 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.253226 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.254800 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.255516 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.257799 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.258196 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.258288 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.262248 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.262254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.262257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.262289 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.264942 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.265918 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.265923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.265925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.266064 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.266566 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.266570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.266572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.266873 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.266987 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.267885 (+ 898us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.268056 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.1.87.124@KRBTEST.COM; defaulting to no policy
I20260619 13:06:03.269840 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.250163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44483 (local address 127.0.0.1:55130)
0619 13:06:03.250930 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.250971 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:03.252732 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:03.253390 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:03.253397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:03.253911 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:03.254627 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.254653 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.255661 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.255670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.257645 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.257663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.258416 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.258427 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.258741 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.259472 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:03.259510 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:03.262020 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:03.265109 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.265122 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.265143 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.265785 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.266184 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.266191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.266197 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.266442 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.267005 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:03.267016 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:03.267491 (+ 475us) client_negotiation.cc:769] Sending connection context
0619 13:06:03.267823 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 13:06:03.268350 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":349,"threads_started":1}
I20260619 13:06:03.271596 24307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.87.124@KRBTEST.COM" created.
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
I20260619 13:06:03.287858 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.272543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36432)
0619 13:06:03.272719 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.272735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:03.272815 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:03.273828 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:03.273836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:03.274342 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:03.274552 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.274566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.276010 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.276017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.277473 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.277490 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.278275 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.278283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.278502 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.279111 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:03.279139 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:03.281938 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:03.285003 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.285011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.285018 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.285524 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.285857 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.285865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.285871 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.286050 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.286619 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:03.286626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:03.286805 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 13:06:03.286960 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 13:06:03.287307 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 13:06:03.287873 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.272831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.1.87.125:34407)
0619 13:06:03.273428 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.273434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.273456 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:06:03.273496 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.273501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.273662 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.273769 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.274745 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.275874 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.277609 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.278112 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.278218 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.282135 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.282146 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.282160 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.282191 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.284872 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.285648 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.285653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.285656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.285757 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.286150 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.286153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.286154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.286488 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.286666 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.287136 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.287486 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":467,"thread_start_us":338,"threads_started":1}
I20260619 13:06:03.290458 24307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:06:03.291683 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.291100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59072)
0619 13:06:03.291299 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.291487 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:06:03.291975 24257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
I20260619 13:06:03.293586 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.293154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59078)
0619 13:06:03.293330 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.293434 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 13:06:03.293902 24307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:38791: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.87.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab.
Entry for principal kudu/127.1.87.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab.
I20260619 13:06:03.304438 1373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskas476m/build/asan/bin/kudu
/tmp/dist-test-taskas476m/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.124:38791
--webserver_interface=127.1.87.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab
--principal=kudu/127.1.87.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.87.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.87.126:44483,127.1.87.125:34407,127.1.87.124:38791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:06:03.343261 24307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } attempt: 1
I20260619 13:06:03.344512 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.343976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59094)
0619 13:06:03.344201 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.344328 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 13:06:03.345484 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.345108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59104)
0619 13:06:03.345269 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.345356 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 13:06:03.345700 24307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:38791: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
I20260619 13:06:03.413947 24307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } attempt: 2
I20260619 13:06:03.415175 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.414645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59106)
0619 13:06:03.414864 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.414973 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 13:06:03.416175 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.415814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59116)
0619 13:06:03.415953 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.416033 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 13:06:03.416409 24307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:38791: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
I20260619 13:06:03.431602 24233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } attempt: 5
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
I20260619 13:06:03.444866 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.432366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36434)
0619 13:06:03.432580 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.432603 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:03.432746 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:03.433106 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:03.433111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:03.433592 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:03.433763 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.433771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.434707 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.434715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.435637 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.435648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.436361 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.436367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.436545 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.437215 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:03.437234 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:03.439485 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:03.442482 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.442489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.442493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.442858 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.443208 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.443213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.443218 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.443406 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.444154 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:03.444158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:03.444297 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 13:06:03.444424 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 13:06:03.444625 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 13:06:03.445107 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.432438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.1.87.125:34407)
0619 13:06:03.432619 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.432627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.432667 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:06:03.432896 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.432900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.432968 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.433068 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.433906 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.434583 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.435775 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.436203 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.436332 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.439676 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.439681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.439684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.439713 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.442255 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.442968 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.442975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.442979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.443096 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.443554 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.443562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.443567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.444020 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.444213 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.444593 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.444820 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:06:03.446094 24233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:06:03.447043 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.446703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59128)
0619 13:06:03.446846 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.446915 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 13:06:03.447885 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.447573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59144)
0619 13:06:03.447721 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.447793 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 13:06:03.448091 24233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:38791: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
I20260619 13:06:03.499361 24233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } attempt: 1
I20260619 13:06:03.500793 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.500047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59148)
0619 13:06:03.500331 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.500542 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 13:06:03.502065 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.501625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59162)
0619 13:06:03.501815 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.501925 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 13:06:03.502765 24233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:38791: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
I20260619 13:06:03.507619 24307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } attempt: 3
I20260619 13:06:03.508766 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.508256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59174)
0619 13:06:03.508479 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.508582 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 13:06:03.510000 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.509512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59186)
0619 13:06:03.509765 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.509856 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260619 13:06:03.510288 24307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:38791: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
I20260619 13:06:03.571139 24233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } attempt: 2
I20260619 13:06:03.572422 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.571826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59200)
0619 13:06:03.572084 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.572211 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 13:06:03.573518 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.573150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59214)
0619 13:06:03.573310 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.573394 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:06:03.573738 24233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:38791: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
W20260619 13:06:03.634063 24314 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:06:03.634359 24314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:06:03.634433 24314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:06:03.634513 24314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:06:03.641542 24314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:06:03.641633 24314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:06:03.641697 24314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:06:03.641749 24314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:06:03.641810 24314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:06:03.651351 24314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.87.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.87.126:44483,127.1.87.125:34407,127.1.87.124:38791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/krb5kdc/kudu_127.1.87.124.keytab
--principal=kudu/127.1.87.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.87.124:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.87.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:06:03.652678 24314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:06:03.653517 24307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } attempt: 4
I20260619 13:06:03.653890 24314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:06:03.654742 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.654198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59220)
0619 13:06:03.654446 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.654547 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 13:06:03.655826 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.655466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59234)
0619 13:06:03.655614 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.655697 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 13:06:03.656051 24307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:38791: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
W20260619 13:06:03.666258 24319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:03.666168 24322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:06:03.666206 24320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:06:03.667384 24314 server_base.cc:1034] running on GCE node
I20260619 13:06:03.668169 24314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:06:03.669633 24314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:06:03.670876 24314 hybrid_clock.cc:648] HybridClock initialized: now 1781874363670790 us; error 64 us; skew 500 ppm
I20260619 13:06:03.671205 24314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:06:03.674379 24314 init.cc:377] Logged in from keytab as kudu/127.1.87.124@KRBTEST.COM (short username kudu)
I20260619 13:06:03.675912 24314 webserver.cc:460] Webserver started at http://127.1.87.124:35823/ using document root <none> and password file <none>
I20260619 13:06:03.676548 24314 fs_manager.cc:359] Metadata directory not provided
I20260619 13:06:03.676637 24314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:06:03.676904 24314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:06:03.679953 24233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } attempt: 3
I20260619 13:06:03.679852 24314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/data/instance:
uuid: "d0d09366702d4d75a1c36cede62c1b0f"
format_stamp: "Formatted at 2026-06-19 13:06:03 on dist-test-slave-tpsm"
server_key: "1c188412286aedd5df43a265aacf4d97"
server_key_iv: "34affd2f008605c000b41b35a043aeea"
server_key_version: "encryptionkey@0"
I20260619 13:06:03.680786 24314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/wal/instance:
uuid: "d0d09366702d4d75a1c36cede62c1b0f"
format_stamp: "Formatted at 2026-06-19 13:06:03 on dist-test-slave-tpsm"
server_key: "1c188412286aedd5df43a265aacf4d97"
server_key_iv: "34affd2f008605c000b41b35a043aeea"
server_key_version: "encryptionkey@0"
I20260619 13:06:03.681118 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.680557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59244)
0619 13:06:03.680785 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.680897 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 13:06:03.682242 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.681797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59258)
0619 13:06:03.681986 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.682084 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260619 13:06:03.682443 24233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.87.124:38791: Network error: Client connection negotiation failed: client connection to 127.1.87.124:38791: connect: Connection refused (error 111)
I20260619 13:06:03.686455 24314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:06:03.690096 24329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:06:03.691766 24314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:06:03.691916 24314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/data,/tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/wal
uuid: "d0d09366702d4d75a1c36cede62c1b0f"
format_stamp: "Formatted at 2026-06-19 13:06:03 on dist-test-slave-tpsm"
server_key: "1c188412286aedd5df43a265aacf4d97"
server_key_iv: "34affd2f008605c000b41b35a043aeea"
server_key_version: "encryptionkey@0"
I20260619 13:06:03.692104 24314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:06:03.706272 24314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:06:03.710701 24314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:06:03.710928 24314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:06:03.729898 24314 rpc_server.cc:307] RPC server started. Bound to: 127.1.87.124:38791
I20260619 13:06:03.729923 24380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.87.124:38791 every 8 connection(s)
I20260619 13:06:03.731197 24314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/data/info.pb
I20260619 13:06:03.734880 24381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:06:03.738811 1373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskas476m/build/asan/bin/kudu as pid 24314
I20260619 13:06:03.739009 1373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskas476m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874004469744-1373-0/minicluster-data/master-2/wal/instance
I20260619 13:06:03.739517 1373 external_mini_cluster.cc:1428] Setting key 3632ae380240c7fff569884f80e567bd
I20260619 13:06:03.741873 24381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
I20260619 13:06:03.760424 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.744436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.1.87.126:44483)
0619 13:06:03.744637 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.744646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.745339 (+ 693us) server_negotiation.cc:407] Connection header received
0619 13:06:03.745598 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.745604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.745683 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.745787 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.747000 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.747795 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.749221 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.749680 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.749815 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.753394 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.753399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.753403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.753439 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.756732 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.757699 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.757704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.757708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.757877 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.758294 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.758302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.758307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.758676 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.758840 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.759757 (+ 917us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.760018 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:06:03.761616 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.744440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.126:44483 (local address 127.0.0.1:55146)
0619 13:06:03.745207 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.745255 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:03.745459 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:03.745851 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:03.745858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:03.746433 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:03.746838 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.746852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.747968 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.747972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.749071 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.749083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.749914 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.749922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.750072 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.750778 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:03.750798 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:03.753192 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:03.756880 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.756894 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.756913 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.757567 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.757995 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.757998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.758000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.758182 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.758796 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:03.758802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:03.759275 (+ 473us) client_negotiation.cc:769] Sending connection context
0619 13:06:03.759636 (+ 361us) client_negotiation.cc:240] Negotiation successful
0619 13:06:03.760132 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":1,"thread_start_us":392,"threads_started":1}
I20260619 13:06:03.761636 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.741540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.1.87.124:38791)
0619 13:06:03.742363 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.742375 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.742419 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:06:03.744185 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.744212 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.744695 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.745074 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.746505 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.747870 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.749101 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.749616 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.749801 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.754138 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.754145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.754159 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.754187 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.757536 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.758343 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.758349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.758351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.758459 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.758875 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.758879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.758881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.759350 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.759667 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.760012 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.760219 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":616,"thread_start_us":365,"threads_started":1}
I20260619 13:06:03.763260 24381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.126@KRBTEST.COM
I20260619 13:06:03.778645 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.764098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.1.87.125:34407)
0619 13:06:03.764277 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.764284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.764306 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:06:03.764407 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.764411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.764493 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.764617 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.765472 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.766685 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.768420 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.769162 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.769318 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.772531 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.772545 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.772552 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.772607 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.775973 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.776819 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.776823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.776825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.776951 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.777399 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.777403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.777405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.777664 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.777776 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.778204 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.778368 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:06:03.778887 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.763987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.125:34407 (local address 127.0.0.1:36442)
0619 13:06:03.764159 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.764182 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:03.764297 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:03.764631 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:03.764634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:03.765087 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:03.765320 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.765334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.766841 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.766845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.768258 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.768272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.769310 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.769314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.769476 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.770104 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:03.770127 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:03.772329 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:03.776110 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.776129 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.776134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.776688 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.777066 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.777073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.777080 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.777275 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.777782 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:03.777790 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:03.777961 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 13:06:03.778113 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 13:06:03.778521 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 13:06:03.779874 24381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:06:03.781836 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.765077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.1.87.126:44483)
0619 13:06:03.765257 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.765265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.765287 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:06:03.765360 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.765368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.765448 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.765531 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.767101 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.768258 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.770028 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.770678 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.770806 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.775639 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.775647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.775651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.775684 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.778887 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.779701 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.779707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.779709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.779878 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.780270 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.780273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.780275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.780614 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.780909 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.781432 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.781613 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.124@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.87.125@KRBTEST.COM
I20260619 13:06:03.793845 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.780516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59276)
0619 13:06:03.780675 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.780693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:03.780797 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:03.781056 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:03.781061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:03.781648 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:03.781832 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.781843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.783072 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.783076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.784107 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.784119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.785100 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.785104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.785254 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.785795 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:03.785812 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:03.787797 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:03.790830 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.790837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.790841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.791375 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.792285 (+ 910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.792291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.792297 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.792454 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.793045 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:03.793050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:03.793219 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 13:06:03.793364 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 13:06:03.793572 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 13:06:03.794142 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.780669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.1.87.124:38791)
0619 13:06:03.780822 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.780826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.780843 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:06:03.780882 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.780885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.780935 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.781007 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.781984 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.782956 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.784240 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.784967 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.785107 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.787975 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.787981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.787987 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.788025 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.790689 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.791492 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.791496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.791498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.791610 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.792590 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.792594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.792596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.792919 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.793031 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.793509 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.793741 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:06:03.800046 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.783533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.1.87.125:34407)
0619 13:06:03.783704 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.783709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.783726 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:06:03.783813 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.783818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.783886 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.783988 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.785474 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.786154 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.787724 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.788436 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.788579 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.793290 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.793294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.793298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.793327 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.797271 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.797979 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.797986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.797990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.798147 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.798653 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.798658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.798661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.798994 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.799284 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.799462 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.799708 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:06:03.800252 24381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f: Bootstrap starting.
I20260619 13:06:03.804339 24381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:03.805751 24381 log.cc:824] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:03.806500 1373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:06:03.808941 24381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f: No bootstrap required, opened a new log
I20260619 13:06:03.809499 24208 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e5f5d0a55d8c40fcb28035b9acce484d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150
I20260619 13:06:03.819753 24381 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } }
I20260619 13:06:03.820175 24381 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d09366702d4d75a1c36cede62c1b0f, State: Initialized, Role: FOLLOWER
I20260619 13:06:03.820683 24381 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } }
I20260619 13:06:03.821650 24387 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:03.821996 24387 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:06:03.822896 24381 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:06:03.831770 24398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:06:03.831909 24398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:06:03.830214 24314 master_runner.cc:186] Error getting master registration for 127.1.87.126:44483: OK, instance_id { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" instance_seqno: 1781874362718352 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:06:03.841691 24314 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:06:03.859687 24233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } attempt: 4
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.126@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
I20260619 13:06:03.876726 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.860937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.1.87.124:38791)
0619 13:06:03.861136 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.861142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.861166 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:06:03.861231 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.861236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.861324 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.861451 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.862219 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.863413 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.865235 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.865989 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.866114 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.870369 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.870374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.870377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.870411 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.873430 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.874250 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.874254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.874256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.874461 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.874942 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.874947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.874950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.875480 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.875657 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.876107 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.876344 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:06:03.876744 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.860437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59278)
0619 13:06:03.860625 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.860648 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:03.860778 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:03.861474 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:03.861478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:03.861857 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:03.862017 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.862025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.863575 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.863581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.865063 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.865082 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.866162 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.866168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.866389 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.867094 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:03.867121 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:03.870180 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:03.873577 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.873583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.873589 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.874130 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.874601 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:03.874607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:03.874611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:03.874790 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:03.875613 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:03.875618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:03.875836 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 13:06:03.876003 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 13:06:03.876317 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 13:06:03.883208 24233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d: Bootstrap starting.
I20260619 13:06:03.887589 24233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:03.889034 24233 log.cc:824] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:03.892238 24233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d: No bootstrap required, opened a new log
I20260619 13:06:03.900043 24233 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } }
I20260619 13:06:03.900729 24233 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f5d0a55d8c40fcb28035b9acce484d, State: Initialized, Role: FOLLOWER
I20260619 13:06:03.901422 24233 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } }
I20260619 13:06:03.902547 24401 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:03.902992 24401 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:06:03.903820 24233 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:06:03.903947 24208 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 13:06:03.904155 24208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:03.907725 24208 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } }
I20260619 13:06:03.910316 24356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d0d09366702d4d75a1c36cede62c1b0f"
I20260619 13:06:03.910797 24356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:03.911190 24208 leader_election.cc:288] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [CANDIDATE]: Term 1 election: Requested vote from peers 35dccef21e5248c0a8f2074818e3c554 (127.1.87.125:34407), d0d09366702d4d75a1c36cede62c1b0f (127.1.87.124:38791)
I20260619 13:06:03.914449 24356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5f5d0a55d8c40fcb28035b9acce484d in term 1.
I20260619 13:06:03.915337 24183 leader_election.cc:302] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0d09366702d4d75a1c36cede62c1b0f, e5f5d0a55d8c40fcb28035b9acce484d; no voters:
I20260619 13:06:03.915987 24401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:06:03.923651 24401 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [term 1 LEADER]: Becoming Leader. State: Replica: e5f5d0a55d8c40fcb28035b9acce484d, State: Running, Role: LEADER
I20260619 13:06:03.924332 24401 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } }
I20260619 13:06:03.939841 24415 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f5d0a55d8c40fcb28035b9acce484d. Latest consensus state: current_term: 1 leader_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:03.940579 24415 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [sys.catalog]: This master's current role is: LEADER
I20260619 13:06:03.955242 24307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } attempt: 5
W20260619 13:06:03.957134 24187 tablet.cc:2369] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:06:03.963078 24335 tablet.cc:2369] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:06:03.953953 24166 master_runner.cc:186] Error getting master registration for 127.1.87.125:34407: OK, instance_id { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" instance_seqno: 1781874363229149 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:06:03.979492 24166 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:06:03.982281 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.925768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.1.87.126:44483)
0619 13:06:03.925913 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.925919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.932083 (+ 6164us) server_negotiation.cc:407] Connection header received
0619 13:06:03.932376 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.932382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.932464 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.932553 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.947471 (+ 14918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.948900 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.953450 (+ 4550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.954443 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.954571 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.973134 (+ 18563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.973142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.973148 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.973212 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:03.977678 (+ 4466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.979689 (+ 2011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.979695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.979698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.979888 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:03.980357 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:03.980364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:03.980374 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:03.980753 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:03.981028 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:03.981605 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 13:06:03.981872 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 13:06:03.985285 24421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:06:03.985412 24421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:06:03.985575 24422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:06:03.988565 24422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 13:06:03 dist-test-slave-tpsm krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.87.125@KRBTEST.COM for kudu/127.1.87.124@KRBTEST.COM
I20260619 13:06:04.011862 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.932292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.1.87.124:38791)
0619 13:06:03.932470 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.932475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.969163 (+ 36688us) server_negotiation.cc:407] Connection header received
0619 13:06:03.969745 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.969755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.969874 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.969968 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.979584 (+ 9616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.981083 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.998922 (+ 17839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.999690 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.999840 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.004310 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:04.004318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:04.004329 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:04.004370 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:04.007897 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:04.009350 (+ 1453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:04.009357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:04.009360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:04.009506 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:04.009962 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:04.009967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:04.009970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:04.010326 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:04.010564 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.011185 (+ 621us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.011475 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:06:04.013217 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.925460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.1.87.125:34407)
0619 13:06:03.925666 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.925675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.953334 (+ 27659us) server_negotiation.cc:407] Connection header received
0619 13:06:03.959974 (+ 6640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.959979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.960072 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.960153 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.971477 (+ 11324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.972787 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.977464 (+ 4677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.978393 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.978523 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.001273 (+ 22750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:04.001280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:04.001284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:04.001322 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:04.006328 (+ 5006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:04.008745 (+ 2417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:04.008751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:04.008756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:04.009060 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:04.009945 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:04.009950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:04.009953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:04.010422 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:04.010631 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.011723 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.012353 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:06:04.020977 24356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [term 1 FOLLOWER]: Refusing update from remote peer e5f5d0a55d8c40fcb28035b9acce484d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:06:04.022018 24401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:06:04.023999 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.956215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.1.87.124:38791)
0619 13:06:03.983525 (+ 27310us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:03.983531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:03.983556 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:06:03.983614 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:03.983620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:03.983718 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:03.984796 (+ 1078us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:03.985048 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.986237 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.988296 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.990676 (+ 2380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.990811 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.996639 (+ 5828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:06:03.996646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:06:03.996661 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:06:03.996701 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:06:04.016287 (+ 19586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:04.017446 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:04.017451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:04.017455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:04.017617 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:06:04.018292 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:06:04.018298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:06:04.018315 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:06:04.018675 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:06:04.018799 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.023258 (+ 4459us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.023511 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27159,"thread_start_us":28201,"threads_started":1}
I20260619 13:06:04.025457 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:03.955845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.87.124:38791 (local address 127.0.0.1:59286)
0619 13:06:03.958595 (+ 2750us) negotiation.cc:107] Waiting for socket to connect
0619 13:06:03.958620 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:06:03.958756 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:06:03.983915 (+ 25159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:06:03.983920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:06:03.984421 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:06:03.984628 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.984641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.986392 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.986397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.987894 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:06:03.987914 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:06:03.990850 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:06:03.990858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:06:03.992396 (+ 1538us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:03.993059 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:06:03.993081 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:06:03.996391 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:06:04.016435 (+ 20044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:04.016440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:04.016444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:04.017243 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:04.017808 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:06:04.017813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:06:04.017837 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:06:04.018078 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:06:04.018972 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:06:04.018978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:06:04.019205 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 13:06:04.024380 (+ 5175us) client_negotiation.cc:240] Negotiation successful
0619 13:06:04.024876 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260619 13:06:04.039829 24387 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f5d0a55d8c40fcb28035b9acce484d. Latest consensus state: current_term: 1 leader_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:04.040146 24387 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:06:04.041177 24307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554: Bootstrap starting.
I20260619 13:06:04.042207 24425 mvcc.cc:205] Tried to move back new op lower bound from 7298557395011182592 to 7298557394640510976. Current Snapshot: MvccSnapshot[applied={T|T < 7298557395011182592}]
I20260619 13:06:04.045274 24307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554: Neither blocks nor log segments found. Creating new log.
I20260619 13:06:04.049106 24307 log.cc:824] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554: Log is configured to *not* fsync() on all Append() calls
I20260619 13:06:04.050204 24415 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:04.050484 24415 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [sys.catalog]: This master's current role is: LEADER
I20260619 13:06:04.050840 24426 mvcc.cc:205] Tried to move back new op lower bound from 7298557395011182592 to 7298557394640510976. Current Snapshot: MvccSnapshot[applied={T|T < 7298557395011182592}]
I20260619 13:06:04.053445 24422 catalog_manager.cc:1347] Generated new cluster ID: 8ca5bdc741df4272b7e517454a3c8cdc
I20260619 13:06:04.053541 24422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:06:04.067549 24387 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:04.067816 24387 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:06:04.081033 24307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554: No bootstrap required, opened a new log
I20260619 13:06:04.083765 24422 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:06:04.085690 24422 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:06:04.102906 24307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } }
I20260619 13:06:04.103694 24307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dccef21e5248c0a8f2074818e3c554, State: Initialized, Role: FOLLOWER
I20260619 13:06:04.104633 24307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } }
I20260619 13:06:04.109974 24422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d: Generated new TSK 0
I20260619 13:06:04.111097 24422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:06:04.115947 24432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:04.116384 24432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:06:04.117354 24307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:06:04.121035 24281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:06:04.123482 24282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "35dccef21e5248c0a8f2074818e3c554"
I20260619 13:06:04.126772 24281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [term 1 FOLLOWER]: Refusing update from remote peer e5f5d0a55d8c40fcb28035b9acce484d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:06:04.130003 24282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5f5d0a55d8c40fcb28035b9acce484d in term 1.
I20260619 13:06:04.132270 24401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:06:04.181726 24432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f5d0a55d8c40fcb28035b9acce484d. Latest consensus state: current_term: 1 leader_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:04.182116 24432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:06:04.181747 24438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:04.182332 24438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 13:06:04.261127 24236 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.243620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.1.87.126:44483)
0619 13:06:04.243907 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.243917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.259083 (+ 15166us) server_negotiation.cc:407] Connection header received
0619 13:06:04.259636 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.259643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.259737 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.259807 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.260894 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55166: BlockingRecv error: recv got EOF from 127.0.0.1:55166 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260619 13:06:04.263619 24311 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.244520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.1.87.125:34407)
0619 13:06:04.244742 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.244748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.261878 (+ 17130us) server_negotiation.cc:407] Connection header received
0619 13:06:04.262270 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.262276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.262356 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.262420 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.263408 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36460: BlockingRecv error: recv got EOF from 127.0.0.1:36460 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 13:06:04.266595 24423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.264704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.1.87.124:38791)
0619 13:06:04.264872 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.264878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.265091 (+ 213us) server_negotiation.cc:407] Connection header received
0619 13:06:04.265339 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.265345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.265435 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.265513 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.266390 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59292: BlockingRecv error: recv got EOF from 127.0.0.1:59292 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:06:04.286684 24456 catalog_manager.cc:1259] Loaded cluster ID: 8ca5bdc741df4272b7e517454a3c8cdc
I20260619 13:06:04.286806 24456 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554: loading cluster ID for follower catalog manager: success
I20260619 13:06:04.291577 24456 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554: acquiring CA information for follower catalog manager: success
I20260619 13:06:04.294119 24456 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35dccef21e5248c0a8f2074818e3c554: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:06:04.322746 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.305014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.1.87.125:34407)
0619 13:06:04.305198 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.305204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.314608 (+ 9404us) server_negotiation.cc:407] Connection header received
0619 13:06:04.315174 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.315182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.315404 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.315548 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.316290 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.317497 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.319777 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.320458 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.320563 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.321297 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.321931 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.322052 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.322316 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.322408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 13:06:04.373616 24423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.360481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.1.87.124:38791)
0619 13:06:04.360666 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.360673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.371627 (+ 10954us) server_negotiation.cc:407] Connection header received
0619 13:06:04.371976 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.371982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.372124 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.372220 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.373331 (+ 1111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59306: BlockingRecv error: recv got EOF from 127.0.0.1:59306 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:06:04.423677 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.409016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.1.87.125:34407)
0619 13:06:04.409254 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.409261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.416574 (+ 7313us) server_negotiation.cc:407] Connection header received
0619 13:06:04.417058 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.417065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.417198 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.417266 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.417955 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.419127 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.421297 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.422028 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.422138 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.422733 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.422906 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.422966 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.423206 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.423296 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 13:06:04.477368 24423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.458104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.1.87.124:38791)
0619 13:06:04.458316 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.458323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.475377 (+ 17054us) server_negotiation.cc:407] Connection header received
0619 13:06:04.475900 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.475907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.476014 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.476093 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.477132 (+ 1039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59308: BlockingRecv error: recv got EOF from 127.0.0.1:59308 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:06:04.508878 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.501447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.1.87.125:34407)
0619 13:06:04.501645 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.501657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.502176 (+ 519us) server_negotiation.cc:407] Connection header received
0619 13:06:04.502383 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.502389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.502501 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.502635 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.503109 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.504340 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.506348 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.507060 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.507191 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.507704 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.507866 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.507970 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.508291 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.508500 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 13:06:04.519027 24423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.516840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.1.87.124:38791)
0619 13:06:04.517065 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.517071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.517456 (+ 385us) server_negotiation.cc:407] Connection header received
0619 13:06:04.517595 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.517600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.517712 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.517824 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.518784 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59320: BlockingRecv error: recv got EOF from 127.0.0.1:59320 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 13:06:04.538390 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.527343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.1.87.125:34407)
0619 13:06:04.527595 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.527603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.532008 (+ 4405us) server_negotiation.cc:407] Connection header received
0619 13:06:04.532247 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.532254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.532353 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.532452 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.532957 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.534159 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.536185 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.536867 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.537000 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.537571 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.537714 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.537807 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.537990 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.538079 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260619 13:06:04.549366 24423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.545562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.1.87.124:38791)
0619 13:06:04.545810 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.545817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.547820 (+ 2003us) server_negotiation.cc:407] Connection header received
0619 13:06:04.548027 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.548033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.548145 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.548252 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.549106 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59334: BlockingRecv error: recv got EOF from 127.0.0.1:59334 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
I20260619 13:06:04.568122 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.561223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.1.87.125:34407)
0619 13:06:04.561414 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.561421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.561933 (+ 512us) server_negotiation.cc:407] Connection header received
0619 13:06:04.562196 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.562204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.562301 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.562414 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.562867 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.564104 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.566024 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.566678 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.566804 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.567271 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.567458 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.567543 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.567710 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.567790 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 13:06:04.576740 24423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.574492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.1.87.124:38791)
0619 13:06:04.574684 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.574690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.575247 (+ 557us) server_negotiation.cc:407] Connection header received
0619 13:06:04.575440 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.575445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.575540 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.575654 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.576489 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59344: BlockingRecv error: recv got EOF from 127.0.0.1:59344 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:06:04.603904 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.596784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.1.87.125:34407)
0619 13:06:04.596981 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.596988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.597531 (+ 543us) server_negotiation.cc:407] Connection header received
0619 13:06:04.597740 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.597747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.597841 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.597934 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.598408 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.599699 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.601673 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.602393 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.602528 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.603043 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.603190 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.603278 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.603462 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.603615 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 13:06:04.613673 24423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.611235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.1.87.124:38791)
0619 13:06:04.611444 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.611450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.611959 (+ 509us) server_negotiation.cc:407] Connection header received
0619 13:06:04.612237 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.612243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.612335 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.612407 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.613428 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59358: BlockingRecv error: recv got EOF from 127.0.0.1:59358 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:06:04.655957 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.649881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.1.87.125:34407)
0619 13:06:04.650131 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.650139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.650520 (+ 381us) server_negotiation.cc:407] Connection header received
0619 13:06:04.650720 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.650728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.650831 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.650947 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.651389 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.652588 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.653889 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.654661 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.654789 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.655146 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.655288 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.655411 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.655513 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.655605 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 13:06:04.663729 24423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.661774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.1.87.124:38791)
0619 13:06:04.661962 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.661968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.662281 (+ 313us) server_negotiation.cc:407] Connection header received
0619 13:06:04.662551 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.662557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.662640 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.662757 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.663490 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59364: BlockingRecv error: recv got EOF from 127.0.0.1:59364 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:06:04.687773 24529 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5f5d0a55d8c40fcb28035b9acce484d" member_type: VOTER last_known_addr { host: "127.1.87.126" port: 44483 } } peers { permanent_uuid: "35dccef21e5248c0a8f2074818e3c554" member_type: VOTER last_known_addr { host: "127.1.87.125" port: 34407 } } peers { permanent_uuid: "d0d09366702d4d75a1c36cede62c1b0f" member_type: VOTER last_known_addr { host: "127.1.87.124" port: 38791 } } }
I20260619 13:06:04.688099 24529 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f5d0a55d8c40fcb28035b9acce484d [sys.catalog]: This master's current role is: LEADER
I20260619 13:06:04.756290 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.749213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.1.87.125:34407)
0619 13:06:04.749422 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.749430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.749995 (+ 565us) server_negotiation.cc:407] Connection header received
0619 13:06:04.750193 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.750198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.750285 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.750391 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.750892 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.752141 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.754137 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.754830 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.754954 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.755490 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.755642 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.755728 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.755897 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.755986 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 13:06:04.764261 24423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:06:04.762287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.1.87.124:38791)
0619 13:06:04.762655 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.762661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.762836 (+ 175us) server_negotiation.cc:407] Connection header received
0619 13:06:04.762978 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.762984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.763079 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.763208 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:06:04.764030 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59368: BlockingRecv error: recv got EOF from 127.0.0.1:59368 (error 108)
Metrics: {"server-negotiator.queue_time_us":222}
I20260619 13:06:04.833796 24398 catalog_manager.cc:1259] Loaded cluster ID: 8ca5bdc741df4272b7e517454a3c8cdc
I20260619 13:06:04.833899 24398 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f: loading cluster ID for follower catalog manager: success
I20260619 13:06:04.836351 24398 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f: acquiring CA information for follower catalog manager: success
I20260619 13:06:04.837966 24398 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0d09366702d4d75a1c36cede62c1b0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:06:04.900301 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.895439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.1.87.125:34407)
0619 13:06:04.895695 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.895701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.895960 (+ 259us) server_negotiation.cc:407] Connection header received
0619 13:06:04.896125 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.896129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.896216 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.896292 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.896722 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.897453 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.898702 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.899102 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.899197 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.899591 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.899712 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.899792 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.899909 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.899968 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 13:06:04.911430 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.906105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.1.87.124:38791)
0619 13:06:04.906310 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.906314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.906677 (+ 363us) server_negotiation.cc:407] Connection header received
0619 13:06:04.906830 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.906834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.906983 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.907086 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.907457 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.908156 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.909382 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.909788 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.909879 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.910323 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.910831 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.910958 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.911059 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.911133 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:06:04.924329 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.917425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.1.87.125:34407)
0619 13:06:04.917660 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.917670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.918441 (+ 771us) server_negotiation.cc:407] Connection header received
0619 13:06:04.918514 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.918521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.918620 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.918720 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.919515 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.920586 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.922257 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.922906 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.923036 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.923559 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.923728 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.923823 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.923960 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.924044 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:06:04.925387 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.918467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.1.87.124:38791)
0619 13:06:04.918617 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.918623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.919524 (+ 901us) server_negotiation.cc:407] Connection header received
0619 13:06:04.919699 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.919704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.919796 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.919895 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.920400 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.921417 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.923248 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.923917 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.924026 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.924648 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.924808 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.924903 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.925067 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.925149 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:06:04.926878 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:06:04.916886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.1.87.126:44483)
0619 13:06:04.917737 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0619 13:06:04.917744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:06:04.917763 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:06:04.917996 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:06:04.918002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:06:04.918234 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:06:04.918408 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:06:04.919419 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.920586 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.924015 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:06:04.924918 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:06:04.925038 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:06:04.925770 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:06:04.926208 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:06:04.926358 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 13:06:04.926462 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 13:06:04.926550 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":784,"threads_started":1}
I20260619 13:06:04.931586 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 24166
I20260619 13:06:04.932473 24228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:05.043939 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 24166
I20260619 13:06:05.065030 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 24240
I20260619 13:06:05.066102 24302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:05.172106 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 24240
I20260619 13:06:05.192605 1373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskas476m/build/asan/bin/kudu pid 24314
I20260619 13:06:05.193735 24376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:06:05.300081 1373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskas476m/build/asan/bin/kudu with pid 24314
2026-06-19T13:06:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4347 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8784 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (360793 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST